org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'

Apache's JIRA Issue Tracker | Paulo Motta | 2 years ago
  1. 0

    I'm trying to copy a large amount of files from HDFS to S3 via distcp and I'm getting the following exception: {code:java} 2015-01-16 20:53:18,187 ERROR [main] org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying hdfs://10.165.35.216/hdfsFolder/file.gz to s3n://s3-bucket/file.gz java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz' at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) 2015-01-16 20:53:18,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz' at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {code} However, when I try hadoop fs -ls s3n://s3-bucket/file.gz the file is there. So probably due to Amazon's S3 eventual consistency the job failure. In my opinion, in order to fix this problem NativeS3FileSystem.getFileStatus must use fs.s3.maxRetries property in order to avoid failures like this.

    Apache's JIRA Issue Tracker | 2 years ago | Paulo Motta
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'
  2. 0

    [HBase-issues] [jira] [Created] (HBASE-9744) Set io.serializations correctly in HFileOutputFormat.configureIncrementalLoad - Grokbase

    grokbase.com | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException
  3. 0

    [Hive-dev] Hive and MR2 - Grokbase

    grokbase.com | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    How do we diagnose a problem within a hunk MR job - Question | Splunk Answers

    splunk.com | 9 months ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException
  6. 0

    [MAPREDUCE-3058] Sometimes task keeps on running while its Syslog says that it is shutdown - ASF JIRA

    apache.org | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.EOFException: Premature EOF: no length prefix available

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.YarnChild

      Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'

      at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus()
    2. Hadoop
      NativeS3FileSystem.getFileStatus
      1. org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445)
      1 frame
    3. Apache Hadoop Distributed Copy
      CopyMapper.map
      1. org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187)
      2. org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233)
      3. org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
      3 frames
    4. Hadoop
      YarnChild$2.run
      1. org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
      2. org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      3. org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      4. org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
      4 frames
    5. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:422)
      2 frames
    6. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      1 frame
    7. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
      1 frame