java.io.IOException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • In a 12 worker nodes setup deployed by serengeti, I tried to run teregen but it failed to complete. The command is: hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen -Dmapred.map.tasks=240 10000000000 /user/joe/terasort-input Got lots of errors like: 12/08/16 03:29:28 INFO mapred.JobClient: Task Id : attempt_201208150845_0006_m_000191_0, Status : FAILED java.io.IOException: Error Recovery for block blk_-3348459061636680999_2713 failed because recovery from primary datanode 192.168.1.88:50010 failed 6 times. Pipeline was 192.168.1.88:50010. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3154) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) attempt_201208150845_0006_m_000191_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient). Then the job failed: 12/08/16 03:36:12 INFO mapred.JobClient: Virtual memory (bytes) snapshot=160994107392 12/08/16 03:36:12 INFO mapred.JobClient: Map input bytes=6874999890 12/08/16 03:36:12 INFO mapred.JobClient: Map output records=6874999890 12/08/16 03:36:12 INFO mapred.JobClient: SPLIT_RAW_BYTES=14437 12/08/16 03:36:12 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201208150845_0006_m_000022 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:352) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) randomtextwriter has the same issue. Binglin helped to take a look and we suspect the open file limitation may be the root cause. I'm trying to enlarge the limitation and re-run the test. Any further suggestions are also appreciated.
    via by Binbin Zhao,
  • In a 12 worker nodes setup deployed by serengeti, I tried to run teregen but it failed to complete. The command is: hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen -Dmapred.map.tasks=240 10000000000 /user/joe/terasort-input Got lots of errors like: 12/08/16 03:29:28 INFO mapred.JobClient: Task Id : attempt_201208150845_0006_m_000191_0, Status : FAILED java.io.IOException: Error Recovery for block blk_-3348459061636680999_2713 failed because recovery from primary datanode 192.168.1.88:50010 failed 6 times. Pipeline was 192.168.1.88:50010. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3154) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) attempt_201208150845_0006_m_000191_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient). Then the job failed: 12/08/16 03:36:12 INFO mapred.JobClient: Virtual memory (bytes) snapshot=160994107392 12/08/16 03:36:12 INFO mapred.JobClient: Map input bytes=6874999890 12/08/16 03:36:12 INFO mapred.JobClient: Map output records=6874999890 12/08/16 03:36:12 INFO mapred.JobClient: SPLIT_RAW_BYTES=14437 12/08/16 03:36:12 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201208150845_0006_m_000022 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:352) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) randomtextwriter has the same issue. Binglin helped to take a look and we suspect the open file limitation may be the root cause. I'm trying to enlarge the limitation and re-run the test. Any further suggestions are also appreciated.
    via by Binbin Zhao,
  • Hadoop. Restart Map
    via Stack Overflow by gmlvsv
    ,
  • error with crawling with nutch
    via Stack Overflow by goodi
    ,
    • java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:352) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    5 more bugmates