java.io.IOException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Copying folders containing + & ! characters between hdfs (using hftp) does not work in distcp For example: Copying folder "string1+string2" at "namenode.address.com", hftp port myport to "/myotherhome/folder" on "myothermachine" does not work myothermachine prompt>>> hadoop --config ~/mycluster/ distcp "hftp://namenode.address.com:myport/myhome/dir/string1+string2" /myotherhome/folder/ -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Error results for hadoop job1: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 08/07/16 00:27:39 INFO tools.DistCp: srcPaths=[hftp://namenode.address.com:myport/myhome/dir/string1+string2] 08/07/16 00:27:39 INFO tools.DistCp: destPath=/myotherhome/folder/ 08/07/16 00:27:41 INFO tools.DistCp: srcCount=2 08/07/16 00:27:42 INFO mapred.JobClient: Running job: job1 08/07/16 00:27:43 INFO mapred.JobClient: map 0% reduce 0% 08/07/16 00:27:58 INFO mapred.JobClient: Task Id : attempt_1_m_000000_0, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:538) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) 08/07/16 00:28:14 INFO mapred.JobClient: Task Id : attempt_1_m_000000_1, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:538) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) 08/07/16 00:28:28 INFO mapred.JobClient: Task Id : attempt_1_m_000000_2, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:538) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) With failures, global counters are inaccurate; consider running with -i Copy failed: java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1053) at org.apache.hadoop.tools.DistCp.copy(DistCp.java:615) at org.apache.hadoop.tools.DistCp.run(DistCp.java:764) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.tools.DistCp.main(DistCp.java:784) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Error log for the map task which failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- INFO org.apache.hadoop.tools.DistCp: FAIL string1+string2/myjobtrackermachine.com-joblog.tar.gz : java.io.IOException: Server returned HTTP response code: 500 for URL: http://mymachine.com:myport/streamFile?filename=/myhome/dir/string1+string2/myjobtrackermachine.com-joblog.tar.gz&ugi=myid,mygroup at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1241) at org.apache.hadoop.dfs.HftpFileSystem.open(HftpFileSystem.java:117) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:371) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:377) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:504) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:279) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    via by Viraj Bhat,
  • Copying folders containing + & ! characters between hdfs (using hftp) does not work in distcp For example: Copying folder "string1+string2" at "namenode.address.com", hftp port myport to "/myotherhome/folder" on "myothermachine" does not work myothermachine prompt>>> hadoop --config ~/mycluster/ distcp "hftp://namenode.address.com:myport/myhome/dir/string1+string2" /myotherhome/folder/ -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Error results for hadoop job1: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 08/07/16 00:27:39 INFO tools.DistCp: srcPaths=[hftp://namenode.address.com:myport/myhome/dir/string1+string2] 08/07/16 00:27:39 INFO tools.DistCp: destPath=/myotherhome/folder/ 08/07/16 00:27:41 INFO tools.DistCp: srcCount=2 08/07/16 00:27:42 INFO mapred.JobClient: Running job: job1 08/07/16 00:27:43 INFO mapred.JobClient: map 0% reduce 0% 08/07/16 00:27:58 INFO mapred.JobClient: Task Id : attempt_1_m_000000_0, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:538) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) 08/07/16 00:28:14 INFO mapred.JobClient: Task Id : attempt_1_m_000000_1, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:538) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) 08/07/16 00:28:28 INFO mapred.JobClient: Task Id : attempt_1_m_000000_2, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:538) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) With failures, global counters are inaccurate; consider running with -i Copy failed: java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1053) at org.apache.hadoop.tools.DistCp.copy(DistCp.java:615) at org.apache.hadoop.tools.DistCp.run(DistCp.java:764) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.tools.DistCp.main(DistCp.java:784) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Error log for the map task which failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- INFO org.apache.hadoop.tools.DistCp: FAIL string1+string2/myjobtrackermachine.com-joblog.tar.gz : java.io.IOException: Server returned HTTP response code: 500 for URL: http://mymachine.com:myport/streamFile?filename=/myhome/dir/string1+string2/myjobtrackermachine.com-joblog.tar.gz&ugi=myid,mygroup at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1241) at org.apache.hadoop.dfs.HftpFileSystem.open(HftpFileSystem.java:117) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:371) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:377) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:504) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:279) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:226) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2208) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    via by Viraj Bhat,
  • Hadoop distcp command not working
    via Stack Overflow by piyush
    ,
  • Using Hadoop distcp to load data into Riak-CS
    via by Sajner, Daniel G,
  • GitHub comment 37#14003273
    via GitHub by pwendell
    ,
  • [HDFS-1109] HFTP and URL Encoding - ASF JIRA
    via by Unknown author,
  • avro error
    via GitHub by priyolahiri
    ,
    • java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1053) at org.apache.hadoop.tools.DistCp.copy(DistCp.java:615) at org.apache.hadoop.tools.DistCp.run(DistCp.java:764) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.tools.DistCp.main(DistCp.java:784)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    5 more bugmates