org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1472884232220_0001_m_000000_0 - exited : java.io.IOException: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.FileNotFoundException: No such file or directory 's3n://hadoopbook/ncdc/all/1901.gz'

Stack Overflow | Simon Zhu | 3 months ago
  1. 0

    Amazon EMR: running Custom Jar with input and output from S3

    Stack Overflow | 3 months ago | Simon Zhu
    org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1472884232220_0001_m_000000_0 - exited : java.io.IOException: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.FileNotFoundException: No such file or directory 's3n://hadoopbook/ncdc/all/1901.gz'
  2. 0

    Unable to run Spectral k-Means Clustering on EMR

    Stack Overflow | 2 years ago | Niko Gamulin
    java.io.FileNotFoundException: No such file or directory 'hdfs://172.31.1.27:9000/user/hadoop/temp/calculations/unitvectors'
  3. 0

    I'm trying to copy a large amount of files from HDFS to S3 via distcp and I'm getting the following exception: {code:java} 2015-01-16 20:53:18,187 ERROR [main] org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying hdfs://10.165.35.216/hdfsFolder/file.gz to s3n://s3-bucket/file.gz java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz' at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) 2015-01-16 20:53:18,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz' at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {code} However, when I try hadoop fs -ls s3n://s3-bucket/file.gz the file is there. So probably due to Amazon's S3 eventual consistency the job failure. In my opinion, in order to fix this problem NativeS3FileSystem.getFileStatus must use fs.s3.maxRetries property in order to avoid failures like this.

    Apache's JIRA Issue Tracker | 2 years ago | Paulo Motta
    java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Read AWS s3 File to Java code

    Stack Overflow | 2 years ago | Edamame
    java.io.FileNotFoundException: s3n:/mybucket/myfile.txt (No such file or directory)
  6. 0

    Apache Flink: "Error triggering a checkpoint as the result of receiving checkpoint barrier"

    Stack Overflow | 5 months ago | OnlyUno
    java.lang.RuntimeException: Error triggering a checkpoint as the result of receiving checkpoint barrier

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.FileNotFoundException

      No such file or directory 's3n://hadoopbook/ncdc/all/1901.gz'

      at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus()
    2. com.amazon.ws
      EmrFileSystem.getFileStatus
      1. com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:818)
      2. com.amazon.ws.emr.hadoop.fs.EmrFileSystem.getFileStatus(EmrFileSystem.java:511)
      2 frames
    3. Apache Hadoop Distributed Copy
      CopyMapper.map
      1. org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
      2. org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
      2 frames
    4. Hadoop
      YarnChild$2.run
      1. org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
      2. org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
      3. org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
      4. org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      4 frames
    5. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:422)
      2 frames
    6. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      1 frame
    7. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      1 frame