org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1376557987570_0005_m_003332_0_spill_0.out

SpringSource Issue Tracker | Wenwu Peng | 3 years ago
  1. 0

    we have a Node have 2 disks, one about 950G and the other about 50G then the 50G disk is full, the 950 disk idle. however, the map job in the Node throw the exception "No space left on device" I will attach more log about resourcemanger when I am in EMC office 13/08/16 02:08:31 INFO mapreduce.Job: map 72% reduce 3% 13/08/16 02:08:33 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_001708_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:08:33 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002215_0, Status : FAILED FSError: java.io.IOException: No space left on device Container killed by the ApplicationMaster. 13/08/16 02:08:34 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002234_0, Status : FAILED FSError: java.io.IOException: No space left on device Container killed by the ApplicationMaster. 13/08/16 02:08:34 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002214_0, Status : FAILED FSError: java.io.IOException: No space left on device Container killed by the ApplicationMaster. 13/08/16 02:08:35 INFO mapreduce.Job: map 73% reduce 3% 13/08/16 02:08:40 INFO mapreduce.Job: map 74% reduce 3% 13/08/16 02:08:46 INFO mapreduce.Job: map 75% reduce 3% 13/08/16 02:08:50 INFO mapreduce.Job: map 76% reduce 3% 13/08/16 02:08:55 INFO mapreduce.Job: map 77% reduce 3% 13/08/16 02:09:00 INFO mapreduce.Job: map 78% reduce 3% 13/08/16 02:09:05 INFO mapreduce.Job: map 79% reduce 3% 13/08/16 02:09:09 INFO mapreduce.Job: map 80% reduce 3% 13/08/16 02:09:10 INFO mapreduce.Job: map 80% reduce 4% 13/08/16 02:09:14 INFO mapreduce.Job: map 81% reduce 4% 13/08/16 02:09:19 INFO mapreduce.Job: map 82% reduce 4% 13/08/16 02:09:25 INFO mapreduce.Job: map 83% reduce 4% 13/08/16 02:09:30 INFO mapreduce.Job: map 84% reduce 4% 13/08/16 02:09:35 INFO mapreduce.Job: map 85% reduce 4% 13/08/16 02:09:39 INFO mapreduce.Job: map 86% reduce 4% 13/08/16 02:09:43 INFO mapreduce.Job: map 87% reduce 4% 13/08/16 02:09:43 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002876_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:43 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003680_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_004417_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003682_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003252_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_004149_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002971_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003977_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:48 INFO mapreduce.Job: map 88% reduce 4% 13/08/16 02:09:53 INFO mapreduce.Job: map 89% reduce 4% 13/08/16 02:09:59 INFO mapreduce.Job: map 90% reduce 4% 13/08/16 02:10:03 INFO mapreduce.Job: map 91% reduce 4% 13/08/16 02:10:07 INFO mapreduce.Job: map 92% reduce 4% 13/08/16 02:10:07 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003211_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:10:07 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_004147_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:10:09 INFO mapreduce.Job: map 92% reduce 5% 13/08/16 02:10:12 INFO mapreduce.Job: map 93% reduce 5% 13/08/16 02:10:16 INFO mapreduce.Job: map 94% reduce 5% 13/08/16 02:10:20 INFO mapreduce.Job: map 95% reduce 5% 13/08/16 02:10:23 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003332_0, Status : FAILED Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1376557987570_0005_m_003332_0_spill_0.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1557) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1451) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:688) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)

    SpringSource Issue Tracker | 3 years ago | Wenwu Peng
    org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1376557987570_0005_m_003332_0_spill_0.out
  2. 0

    we have a Node have 2 disks, one about 950G and the other about 50G then the 50G disk is full, the 950 disk idle. however, the map job in the Node throw the exception "No space left on device" I will attach more log about resourcemanger when I am in EMC office 13/08/16 02:08:31 INFO mapreduce.Job: map 72% reduce 3% 13/08/16 02:08:33 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_001708_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:08:33 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002215_0, Status : FAILED FSError: java.io.IOException: No space left on device Container killed by the ApplicationMaster. 13/08/16 02:08:34 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002234_0, Status : FAILED FSError: java.io.IOException: No space left on device Container killed by the ApplicationMaster. 13/08/16 02:08:34 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002214_0, Status : FAILED FSError: java.io.IOException: No space left on device Container killed by the ApplicationMaster. 13/08/16 02:08:35 INFO mapreduce.Job: map 73% reduce 3% 13/08/16 02:08:40 INFO mapreduce.Job: map 74% reduce 3% 13/08/16 02:08:46 INFO mapreduce.Job: map 75% reduce 3% 13/08/16 02:08:50 INFO mapreduce.Job: map 76% reduce 3% 13/08/16 02:08:55 INFO mapreduce.Job: map 77% reduce 3% 13/08/16 02:09:00 INFO mapreduce.Job: map 78% reduce 3% 13/08/16 02:09:05 INFO mapreduce.Job: map 79% reduce 3% 13/08/16 02:09:09 INFO mapreduce.Job: map 80% reduce 3% 13/08/16 02:09:10 INFO mapreduce.Job: map 80% reduce 4% 13/08/16 02:09:14 INFO mapreduce.Job: map 81% reduce 4% 13/08/16 02:09:19 INFO mapreduce.Job: map 82% reduce 4% 13/08/16 02:09:25 INFO mapreduce.Job: map 83% reduce 4% 13/08/16 02:09:30 INFO mapreduce.Job: map 84% reduce 4% 13/08/16 02:09:35 INFO mapreduce.Job: map 85% reduce 4% 13/08/16 02:09:39 INFO mapreduce.Job: map 86% reduce 4% 13/08/16 02:09:43 INFO mapreduce.Job: map 87% reduce 4% 13/08/16 02:09:43 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002876_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:43 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003680_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_004417_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003682_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003252_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_004149_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_002971_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:44 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003977_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:09:48 INFO mapreduce.Job: map 88% reduce 4% 13/08/16 02:09:53 INFO mapreduce.Job: map 89% reduce 4% 13/08/16 02:09:59 INFO mapreduce.Job: map 90% reduce 4% 13/08/16 02:10:03 INFO mapreduce.Job: map 91% reduce 4% 13/08/16 02:10:07 INFO mapreduce.Job: map 92% reduce 4% 13/08/16 02:10:07 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003211_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:10:07 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_004147_0, Status : FAILED FSError: java.io.IOException: No space left on device 13/08/16 02:10:09 INFO mapreduce.Job: map 92% reduce 5% 13/08/16 02:10:12 INFO mapreduce.Job: map 93% reduce 5% 13/08/16 02:10:16 INFO mapreduce.Job: map 94% reduce 5% 13/08/16 02:10:20 INFO mapreduce.Job: map 95% reduce 5% 13/08/16 02:10:23 INFO mapreduce.Job: Task Id : attempt_1376557987570_0005_m_003332_0, Status : FAILED Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1376557987570_0005_m_003332_0_spill_0.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1557) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1451) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:688) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)

    SpringSource Issue Tracker | 3 years ago | Wenwu Peng
    org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1376557987570_0005_m_003332_0_spill_0.out
  3. 0

    Hadoop crashed while running terasort?

    Stack Overflow | 2 years ago
    org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1429766544852_0001_m_001255_0_spill_1.out
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Apache Hadoop user mailing list

    gmane.org | 1 year ago
    org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1432720271082_0005_m_000045_0_spill_0.out
  6. 0

    Hadoop常见错误及解决办法汇总 - 突破 - 博客频道 - CSDN.NET

    csdn.net | 1 year ago
    org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1399539856880_0016_m_000029_2_spill_0.out

    3 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.util.DiskChecker$DiskErrorException

      Could not find any valid local directory for attempt_1376557987570_0005_m_003332_0_spill_0.out

      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite()
    2. Hadoop
      LocalDirAllocator.getLocalPathForWrite
      1. org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
      2. org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
      3. org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
      3 frames
    3. Hadoop
      YarnChild$2.run
      1. org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
      2. org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1557)
      3. org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1451)
      4. org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:688)
      5. org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
      6. org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
      7. org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
      7 frames
    4. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:396)
      2 frames
    5. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
      1 frame
    6. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
      1 frame