java.lang.RuntimeException: Error triggering a checkpoint as the result of receiving checkpoint barrier

Stack Overflow | OnlyUno | 8 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Apache Flink: "Error triggering a checkpoint as the result of receiving checkpoint barrier"

    Stack Overflow | 8 months ago | OnlyUno
    java.lang.RuntimeException: Error triggering a checkpoint as the result of receiving checkpoint barrier
  2. 0

    Copy Task fails using DistCp

    Stack Overflow | 2 years ago | user2664210
    java.io.FileNotFoundException: No such file or directory 'S3-DEST_PATH/log20141106-132606702+0000.53943299650332301.0000001520141106-132606702+0000.53943299650332301.00000015.avro'
  3. 0

    AWS Developer Forums: Using Hadoop's DistributedCache ...

    amazon.com | 2 years ago
    java.io.FileNotFoundException: No such file or directory '/path/to/archive.zip#directory'
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I'm trying to copy a large amount of files from HDFS to S3 via distcp and I'm getting the following exception: {code:java} 2015-01-16 20:53:18,187 ERROR [main] org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying hdfs://10.165.35.216/hdfsFolder/file.gz to s3n://s3-bucket/file.gz java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz' at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) 2015-01-16 20:53:18,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz' at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {code} However, when I try hadoop fs -ls s3n://s3-bucket/file.gz the file is there. So probably due to Amazon's S3 eventual consistency the job failure. In my opinion, in order to fix this problem NativeS3FileSystem.getFileStatus must use fs.s3.maxRetries property in order to avoid failures like this.

    Apache's JIRA Issue Tracker | 2 years ago | Paulo Motta
    java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'

    Root Cause Analysis

    1. java.io.FileNotFoundException

      No such file or directory 's3n://xxxx/flink/datum/checkpoints/5bdceb1b79bd568de16fe82b01887b96/chk-14046/ee6d12f7-9b96-4f86-b66a-00ccdd23a8cc'

      at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus()
    2. Hadoop
      NativeS3FileSystem.getFileStatus
      1. org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:507)
      1 frame
    3. flink-runtime
      HadoopFileSystem.getFileStatus
      1. org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
      1 frame
    4. org.apache.flink
      StreamTaskStateList.getStateSize
      1. org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
      2. org.apache.flink.runtime.state.filesystem.AbstractFsStateSnapshot.getStateSize(AbstractFsStateSnapshot.java:134)
      3. org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:87)
      3 frames
    5. flink-runtime
      RuntimeEnvironment.acknowledgeCheckpoint
      1. org.apache.flink.runtime.taskmanager.RuntimeEnvironment.acknowledgeCheckpoint(RuntimeEnvironment.java:231)
      1 frame
    6. org.apache.flink
      StreamTask.invoke
      1. org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:528)
      2. org.apache.flink.streaming.runtime.tasks.StreamTask$2.onEvent(StreamTask.java:695)
      3. org.apache.flink.streaming.runtime.tasks.StreamTask$2.onEvent(StreamTask.java:691)
      4. org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
      5. org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
      6. org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:175)
      7. org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:65)
      8. org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
      8 frames
    7. flink-runtime
      Task.run
      1. org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
      1 frame
    8. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)
      1 frame