java.io.FileNotFoundException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Hadoop 2.4.1 Release Notes
    via by Unknown author,
  • mongo-hadoop and hadoop-0.23.1
    via by Mark Lewandowski,
  • [mongodb-user] mongo-hadoop and hadoop-0.23.1
    via by Mark Lewandowski,
  • Hadoop 2.6.0 Release Notes
    via by Unknown author,
  • Hadoop file not found exception
    via Stack Overflow by daivik
    ,
  • Spark streaming from multiple folder
    via Stack Overflow by Cheeko
    ,
  • Hi guys, facing the similar type of issue in my cluster. when running examples in local mode executed successfully. while running same code on cdh5 spark cluster not work at all. sparkR pkgs distributed in all nodes and tested locally. {color:red} Environment details: {color} <b>user@master-server:~$ java -version</b> java version "1.6.0_32" user@master-server:~$ hadoop version Hadoop 2.3.0-cdh5.1.3 Subversion git://github.sf.cloudera.com/CDH/cdh.git -r 8e266e052e423af592871e2dfe09d54c03f6a0e8 Compiled by jenkins on 2014-09-17T02:20Z Compiled with protoc 2.5.0 From source with checksum ec11b8ec19ca2bf3e7cb1bbe4ee182 This command was run using /usr/lib/hadoop/hadoop-common-2.3.0-cdh5.1.3.jar <b>user@master-server:~$ spark-shell</b> 14/10/14 14:59:09 INFO SecurityManager: Changing view acls to: user 14/10/14 14:59:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user) 14/10/14 14:59:09 INFO HttpServer: Starting HTTP Server Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.0.0 {color:red} pom.xml file {color} for hadoop version and spark version <!-- NOTE: Adding the hadoop dependency first ensures that we pull in protobuf from Hadoop before Spark --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.3.0-mr1-cdh5.1.3</version> . . . <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.0.0</version> <b>properties</b> : <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.6</java.version> <scala.version>2.10.4</scala.version> <scala.binary.version>2.10</scala.binary.version> <hadoop.version>2.3.0-mr1-cdh5.1.3</hadoop.version> Also added this repository : <repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository> in ./install.sh file changes hadoop and spark version as per pom.xml file : SPARK_HADOOP_VERSION ?= 2.3.0-mr1-cdh5.1.3 SPARK_VERSION ?= 1.0.0 {color:red} Logs: {color} h5. 1. from R console : * > options( java.parameters = "-Xmx1g" ) > library("devtools") > library("SparkR") Loading required package: rJava [SparkR] Initializing with classpath /usr/lib/R/site-library/SparkR/sparkr-assembly-0.1.jar > sc <- sparkR.init(master="spark://master-server:7077","SparkRinCluster") 14/10/14 14:30:07 INFO Slf4jLogger: Slf4jLogger started 14/10/14 14:30:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java c lasses where applicable > lines <- textFile(sc, "/home/user/temp.txt") words <- flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <- lapply(words, function(word) { list(word, 1L) }) counts <- reduceByKey(wordCount, "+", 2L) output <- collect(counts)> > words <- flatMap(lines, + function(line) { + strsplit(line, " ")[[1]] + }) > wordCount <- lapply(words, function(word) { list(word, 1L) }) > > counts <- reduceByKey(wordCount, "+", 2L) > output <- collect(counts) 14/10/14 14:30:30 INFO FileInputFormat: Total input paths to process : 1 14/10/14 14:30:45 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure t hat workers are registered and have sufficient memory 14/10/14 14:31:00 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered a nd have sufficient memory 14/10/14 14:31:13 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up. Error in .jcall(getJRDD(rdd), "Ljava/util/List;", "collect") : org.apache.spark.SparkException: Job aborted due to stage failure: All masters are unresponsive! Giving up. > 14/10/14 14:31:14 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 14/10/14 14:31:14 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 14/10/14 14:31:14 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. ~~~~~~~~~~~~ h5. 2.Spark logs from /var/log/spark/spark-master-....log * ~~~~~~~~~~~~~~ 2014-10-07 17:27:27,568 ERROR akka.remote.EndpointWriter: AssociationError [akka.tcp://user@spark-master:7077] -> [akka.tcp://spark@spark-master:42718]: Error [Association failed with [akka.tcp://spark@spark-master:42718]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@spark-master:42718] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: spark-master/143.96.25.30:42718 ] 2014-10-07 17:27:27,616 ERROR org.apache.spark.scheduler.EventLoggingListener: Exception in parsing logging info from directory /user/spark/applicationHistory/hdfstest-1412656046141 java.io.FileNotFoundException: File /user/spark/applicationHistory/hdfstest-1412656046141 does not exist at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:362) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1483) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1523) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:564) at org.apache.spark.scheduler.EventLoggingListener$.parseLoggingInfo(EventLoggingListener.scala:182) at org.apache.spark.scheduler.EventLoggingListener$.parseLoggingInfo(EventLoggingListener.scala:219) at org.apache.spark.deploy.master.Master.rebuildSparkUI(Master.scala:675) at org.apache.spark.deploy.master.Master.removeApplication(Master.scala:648) at org.apache.spark.deploy.master.Master.finishApplication(Master.scala:626) at org.apache.spark.deploy.master.Master$$anonfun$receive$1$$anonfun$applyOrElse$26.apply(Master.scala:385) at org.apache.spark.deploy.master.Master$$anonfun$receive$1$$anonfun$applyOrElse$26.apply(Master.scala:385) at scala.Option.foreach(Option.scala:236) at org.apache.spark.deploy.master.Master$$anonfun$receive$1.applyOrElse(Master.scala:385) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2014-10-07 17:27:27,617 WARN org.apache.spark.deploy.master.Master: Application HdfsTest (app-20141007172727-0001) has no valid logs: /user/spark/applicationHistory/hdfstest-1412656046141 2014-10-07 17:27:27,618 INFO org.apache.spark.deploy.master.Master: akka.tcp://spark@spark-master:42718 got disassociated, removing it. 2014-10-07 17:27:27,618 INFO org.apache.spark.deploy.master.Master: akka.tcp://spark@spark-master:42718 got disassociated, removing it. 2014-10-07 17:27:27,618 INFO org.apache.spark.deploy.master.Master: akka.tcp://spark@spark-master:42718 got disassociated, removing it. 2014-10-07 17:27:27,618 INFO org.apache.spark.deploy.master.Master: akka.tcp://spark@spark-master:42718 got disassociated, removing it. 2014-10-07 17:27:27,631 WARN org.apache.spark.deploy.master.Master: Got status update for unknown executor app-20141007172727-0001/1 2014-10-07 17:27:27,632 WARN org.apache.spark.deploy.master.Master: Got status update for unknown executor app-20141007172727-0001/2 2014-10-07 17:27:27,633 WARN org.apache.spark.deploy.master.Master: Got status update for unknown executor app-20141007172727-0001/0 2014-10-07 17:27:27,635 WARN org.apache.spark.deploy.master.Master: Got status update for unknown executor app-20141007172727-0001/3 2014-10-08 09:15:16,685 INFO org.apache.spark.deploy.master.Master: Registering app HdfsTest 2014-10-08 09:15:16,686 INFO org.apache.spark.deploy.master.Master: Registered app HdfsTest with ID app-20141008091516-0002 2014-10-08 09:15:16,686 INFO org.apache.spark.deploy.master.Master: Launching executor app-20141008091516-0002/0 on worker worker-20141007160129-spark-worker-7078 2014-10-08 09:15:16,686 INFO org.apache.spark.deploy.master.Master: Launching executor app-20141008091516-0002/1 on worker worker-20141007160129-spark-worker-7078 2014-10-08 09:15:16,686 INFO org.apache.spark.deploy.master.Master: Launching executor app-20141008091516-0002/2 on worker worker-20141007160129-spark-worker-7078 2014-10-08 09:15:16,686 INFO org.apache.spark.deploy.master.Master: Launching executor app-20141008091516-0002/3 on worker worker-20141007160130-spark-worker-7078 2014-10-08 09:15:17,120 INFO org.apache.spark.deploy.master.Master: akka.tcp://spark@spark-master:50150 got disassociated, removing it. Please help me to debug the issue and make it running.
    via by Harihar Nahak,
    • java.io.FileNotFoundException: File file:/Users/anshumahajan/part-r-* does not exist at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:371) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1483) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1523) at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:1680) at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1679) at org.apache.hadoop.fs.ChecksumFileSystem.listLocatedStatus(ChecksumFileSystem.java:583) at org.apache.hadoop.hive.shims.Hadoop23Shims.listLocatedStatus(Hadoop23Shims.java:667) at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:361) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:634) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:620) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

    Users with the same issue

    r1chjames
    7 times, last one,
    Unknown visitor1 times, last one,