java.lang.NoSuchMethodException: org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()

Apache's JIRA Issue Tracker | Staffan Arvidsson | 2 years ago
  1. 0

    Apache Spark User List - Issues when combining Spark and a third party java library

    nabble.com | 1 year ago
    java.lang.NoSuchMethodException: org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()
  2. 0

    I'm using Maven and Eclipse to build my project. When I import the CDK (https://github.com/egonw/cdk) jar-files that I need, and setup the SparkContext and try for instance reading a file (by simply "val lines = sc.textFile(filePath)") I get the following errors in the log: {quote} [main] DEBUG org.apache.spark.rdd.HadoopRDD - SplitLocationInfo and other new Hadoop classes are unavailable. Using the older Hadoop location info code. java.lang.ClassNotFoundException: org.apache.hadoop.mapred.InputSplitWithLocationInfo at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:191) at org.apache.spark.rdd.HadoopRDD$SplitInfoReflections.<init>(HadoopRDD.scala:381) at org.apache.spark.rdd.HadoopRDD$.liftedTree1$1(HadoopRDD.scala:391) at org.apache.spark.rdd.HadoopRDD$.<init>(HadoopRDD.scala:390) at org.apache.spark.rdd.HadoopRDD$.<clinit>(HadoopRDD.scala) at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:159) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) at org.apache.spark.rdd.RDD.foreach(RDD.scala:765) {quote} later in the log: {quote} [Executor task launch worker-0] DEBUG org.apache.spark.deploy.SparkHadoopUtil - Couldn't find method for retrieving thread-level FileSystem input data java.lang.NoSuchMethodException: org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics() at java.lang.Class.getDeclaredMethod(Class.java:2009) at org.apache.spark.util.Utils$.invoke(Utils.scala:1733) at org.apache.spark.deploy.SparkHadoopUtil$$anonfun$getFileSystemThreadStatistics$1.apply(SparkHadoopUtil.scala:178) at org.apache.spark.deploy.SparkHadoopUtil$$anonfun$getFileSystemThreadStatistics$1.apply(SparkHadoopUtil.scala:178) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.deploy.SparkHadoopUtil.getFileSystemThreadStatistics(SparkHadoopUtil.scala:178) at org.apache.spark.deploy.SparkHadoopUtil.getFSBytesReadOnThreadCallback(SparkHadoopUtil.scala:138) at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:220) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {quote} There has also been issues related to "HADOOP_HOME" not being set etc., but which seems to be intermittent and only occur sometimes. After testing different versions of both CDK and Spark, I've found out that the Spark version 0.9.1 seems to get things to work. This will not solve my problem though, as I will later need to use functionality from the MLlib that are only in the newer versions of Spark.

    Apache's JIRA Issue Tracker | 2 years ago | Staffan Arvidsson
    java.lang.NoSuchMethodException: org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()
  3. 0

    Apache Spark User List - output folder structure not getting commited and remains as _temporary

    nabble.com | 1 year ago
    java.lang.NoSuchMethodException: org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Spark, mail # user - Spark Job Hangs on our production cluster - 2015-08-11, 20:19

    search-hadoop.com | 1 year ago
    java.lang.NoSuchMethodException: org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()
  6. 0

    Error when creating Project

    GitHub | 2 years ago | itonics-tbeauvais
    java.lang.NoSuchMethodException: hudson.model.AbstractProject.convertUpstreamBuildTrigger(java.util.Set)

  1. tyson925 9 times, last 6 months ago
  2. davidvanlaatum 4 times, last 7 days ago
  3. franky li 19 times, last 1 month ago
  4. pranavprakash 2 times, last 1 month ago
  5. ank_07 1 times, last 4 months ago
11 more registered users
27 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.lang.NoSuchMethodException

    org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()

    at java.lang.Class.getDeclaredMethod()
  2. Java RT
    Class.getDeclaredMethod
    1. java.lang.Class.getDeclaredMethod(Class.java:2009)
    1 frame
  3. Spark
    SparkHadoopUtil$$anonfun$getFileSystemThreadStatistics$1.apply
    1. org.apache.spark.util.Utils$.invoke(Utils.scala:1733)
    2. org.apache.spark.deploy.SparkHadoopUtil$$anonfun$getFileSystemThreadStatistics$1.apply(SparkHadoopUtil.scala:178)
    3. org.apache.spark.deploy.SparkHadoopUtil$$anonfun$getFileSystemThreadStatistics$1.apply(SparkHadoopUtil.scala:178)
    3 frames
  4. Scala
    AbstractTraversable.map
    1. scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    2. scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    3. scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    4. scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    5. scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    6. scala.collection.AbstractTraversable.map(Traversable.scala:105)
    6 frames
  5. Spark
    Executor$TaskRunner.run
    1. org.apache.spark.deploy.SparkHadoopUtil.getFileSystemThreadStatistics(SparkHadoopUtil.scala:178)
    2. org.apache.spark.deploy.SparkHadoopUtil.getFSBytesReadOnThreadCallback(SparkHadoopUtil.scala:138)
    3. org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:220)
    4. org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210)
    5. org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
    6. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
    7. org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
    8. org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
    9. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
    10. org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
    11. org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    12. org.apache.spark.scheduler.Task.run(Task.scala:56)
    13. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
    13 frames
  6. Java RT
    Thread.run
    1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    3. java.lang.Thread.run(Thread.java:745)
    3 frames