java.io.IOException: Job failed!

SpringSource Issue Tracker | Yifeng Xiao | 4 years ago
  1. 0

    Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Job failed!
  2. 0

    Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Job failed!
  3. 0

    Create a large scale cluster: {code} cluster name: ha_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------------------------------------- ComputeMaster [mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_metrics] 3 2 2048 SHARED 20 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- ha_mapr-ComputeMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.137 Service Ready ha_mapr-ComputeMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.136 Service Ready ha_mapr-ComputeMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.138 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- Monitor [mapr_zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------- ha_mapr-Monitor-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.152 Service Ready ha_mapr-Monitor-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.149 Service Ready ha_mapr-Monitor-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.148 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 5 2 2048 LOCAL 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-worker-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.145 Service Ready ha_mapr-worker-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.147 Service Ready ha_mapr-worker-4 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.144 Service Ready ha_mapr-worker-3 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready ha_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.143 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 2048 SHARED 30 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-client-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.151 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------- DataMaster [mapr_cldb, mapr_nfs, mapr_fileserver] 3 2 2048 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ ha_mapr-DataMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.135 Service Ready ha_mapr-DataMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.140 Service Ready ha_mapr-DataMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- ha_mapr-mysqlserver-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.150 Service Ready {code} log on client vm and run wordcout job: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvol/input/ /myvol/output/ 13/02/01 01:56:18 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:56:18 INFO input.FileInputFormat: Total input paths to process : 1 13/02/01 01:56:18 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's output directory at /myvol/output 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's user history location directory at /myvol/output/_logs 13/02/01 01:56:18 INFO mapred.JobClient: Running job: job_201301310814_0004 13/02/01 01:56:19 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:56:27 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0004_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:31 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_2, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:43 INFO mapred.JobClient: Job job_201301310814_0004 failed with state FAILED due to: NA 13/02/01 01:56:43 INFO mapred.JobClient: Counters: 8 13/02/01 01:56:43 INFO mapred.JobClient: Job Counters 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=9826 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Rack-local map tasks=1 13/02/01 01:56:43 INFO mapred.JobClient: Launched map tasks=4 13/02/01 01:56:43 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Failed map tasks=1 {code} Benchmark: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar terasort /myvol/terasort-input /myvol/terasort-output 13/02/01 01:53:39 INFO terasort.TeraSort: starting 13/02/01 01:53:39 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:39 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:53:40 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/02/01 01:53:40 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/02/01 01:53:40 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:53:41 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's output directory at maprfs:/myvol/terasort-output 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/myvol/terasort-output/_logs 13/02/01 01:53:41 INFO mapred.JobClient: Running job: job_201301310814_0003 13/02/01 01:53:42 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0003_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0003_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0003_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0003_m_000001_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:54:04 INFO mapred.JobClient: Job job_201301310814_0003 failed with state FAILED due to: NA 13/02/01 01:54:04 INFO mapred.JobClient: Counters: 8 13/02/01 01:54:04 INFO mapred.JobClient: Job Counters 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=15134 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Rack-local map tasks=6 13/02/01 01:54:04 INFO mapred.JobClient: Launched map tasks=8 13/02/01 01:54:04 INFO mapred.JobClient: Data-local map tasks=2 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/02/01 01:51:18 INFO fs.TestDFSIO: nrFiles = 5 13/02/01 01:51:18 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/02/01 01:51:18 INFO fs.TestDFSIO: bufferSize = 1000000 13/02/01 01:51:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/02/01 01:51:19 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 13/02/01 01:51:19 INFO fs.TestDFSIO: created control files for: 5 files 13/02/01 01:51:19 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:51:20 INFO mapred.FileInputFormat: Total input paths to process : 5 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs 13/02/01 01:51:20 INFO mapred.JobClient: Running job: job_201301310814_0001 13/02/01 01:51:21 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:51:30 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_0, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:35 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000002_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_0, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000004_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000003_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_1, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_1, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000002_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_1, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000004_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000003_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:43 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_2, Status : FAILED on node 10.111.88.147 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_2, Status : FAILED on node 10.111.88.145 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_2, Status : FAILED on node 10.111.88.144 Error: Java heap space 13/02/01 01:51:46 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000004_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:47 INFO mapred.JobClient: Job job_201301310814_0001 failed with state FAILED due to: NA 13/02/01 01:51:47 INFO mapred.JobClient: Counters: 8 13/02/01 01:51:47 INFO mapred.JobClient: Job Counters 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=35682 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Rack-local map tasks=16 13/02/01 01:51:47 INFO mapred.JobClient: Launched map tasks=19 13/02/01 01:51:47 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Job failed!
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Create a large scale cluster: {code} cluster name: ha_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------------------------------------- ComputeMaster [mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_metrics] 3 2 2048 SHARED 20 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- ha_mapr-ComputeMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.137 Service Ready ha_mapr-ComputeMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.136 Service Ready ha_mapr-ComputeMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.138 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- Monitor [mapr_zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------- ha_mapr-Monitor-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.152 Service Ready ha_mapr-Monitor-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.149 Service Ready ha_mapr-Monitor-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.148 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 5 2 2048 LOCAL 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-worker-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.145 Service Ready ha_mapr-worker-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.147 Service Ready ha_mapr-worker-4 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.144 Service Ready ha_mapr-worker-3 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready ha_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.143 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 2048 SHARED 30 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-client-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.151 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------- DataMaster [mapr_cldb, mapr_nfs, mapr_fileserver] 3 2 2048 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ ha_mapr-DataMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.135 Service Ready ha_mapr-DataMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.140 Service Ready ha_mapr-DataMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- ha_mapr-mysqlserver-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.150 Service Ready {code} log on client vm and run wordcout job: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvol/input/ /myvol/output/ 13/02/01 01:56:18 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:56:18 INFO input.FileInputFormat: Total input paths to process : 1 13/02/01 01:56:18 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's output directory at /myvol/output 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's user history location directory at /myvol/output/_logs 13/02/01 01:56:18 INFO mapred.JobClient: Running job: job_201301310814_0004 13/02/01 01:56:19 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:56:27 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0004_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:31 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_2, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:43 INFO mapred.JobClient: Job job_201301310814_0004 failed with state FAILED due to: NA 13/02/01 01:56:43 INFO mapred.JobClient: Counters: 8 13/02/01 01:56:43 INFO mapred.JobClient: Job Counters 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=9826 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Rack-local map tasks=1 13/02/01 01:56:43 INFO mapred.JobClient: Launched map tasks=4 13/02/01 01:56:43 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Failed map tasks=1 {code} Benchmark: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar terasort /myvol/terasort-input /myvol/terasort-output 13/02/01 01:53:39 INFO terasort.TeraSort: starting 13/02/01 01:53:39 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:39 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:53:40 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/02/01 01:53:40 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/02/01 01:53:40 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:53:41 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's output directory at maprfs:/myvol/terasort-output 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/myvol/terasort-output/_logs 13/02/01 01:53:41 INFO mapred.JobClient: Running job: job_201301310814_0003 13/02/01 01:53:42 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0003_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0003_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0003_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0003_m_000001_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:54:04 INFO mapred.JobClient: Job job_201301310814_0003 failed with state FAILED due to: NA 13/02/01 01:54:04 INFO mapred.JobClient: Counters: 8 13/02/01 01:54:04 INFO mapred.JobClient: Job Counters 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=15134 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Rack-local map tasks=6 13/02/01 01:54:04 INFO mapred.JobClient: Launched map tasks=8 13/02/01 01:54:04 INFO mapred.JobClient: Data-local map tasks=2 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/02/01 01:51:18 INFO fs.TestDFSIO: nrFiles = 5 13/02/01 01:51:18 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/02/01 01:51:18 INFO fs.TestDFSIO: bufferSize = 1000000 13/02/01 01:51:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/02/01 01:51:19 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 13/02/01 01:51:19 INFO fs.TestDFSIO: created control files for: 5 files 13/02/01 01:51:19 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:51:20 INFO mapred.FileInputFormat: Total input paths to process : 5 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs 13/02/01 01:51:20 INFO mapred.JobClient: Running job: job_201301310814_0001 13/02/01 01:51:21 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:51:30 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_0, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:35 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000002_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_0, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000004_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000003_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_1, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_1, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000002_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_1, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000004_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000003_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:43 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_2, Status : FAILED on node 10.111.88.147 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_2, Status : FAILED on node 10.111.88.145 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_2, Status : FAILED on node 10.111.88.144 Error: Java heap space 13/02/01 01:51:46 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000004_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:47 INFO mapred.JobClient: Job job_201301310814_0001 failed with state FAILED due to: NA 13/02/01 01:51:47 INFO mapred.JobClient: Counters: 8 13/02/01 01:51:47 INFO mapred.JobClient: Job Counters 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=35682 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Rack-local map tasks=16 13/02/01 01:51:47 INFO mapred.JobClient: Launched map tasks=19 13/02/01 01:51:47 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Job failed!
  6. 0

    getting Null Pointer Exception while doing Secondary sort

    Stack Overflow | 3 years ago | user1585111
    java.io.IOException: Job failed!

    9 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Job failed!

      at org.apache.hadoop.mapred.JobClient.runJob()
    2. Hadoop
      JobClient.runJob
      1. org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
      1 frame
    3. Apache Hadoop MapReduce Examples
      TeraSort.run
      1. org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248)
      1 frame
    4. Hadoop
      ToolRunner.run
      1. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      1 frame
    5. Apache Hadoop MapReduce Examples
      TeraSort.main
      1. org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257)
      1 frame
    6. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    7. Hadoop
      ProgramDriver.driver
      1. org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      2. org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      2 frames
    8. Apache Hadoop MapReduce Examples
      ExampleDriver.main
      1. org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
      1 frame
    9. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    10. Hadoop
      RunJar.main
      1. org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      1 frame