java.io.IOException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}
    via by Yifeng Xiao,
  • Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}
    via by Yifeng Xiao,
  • Create a large scale cluster: {code} cluster name: ha_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------------------------------------- ComputeMaster [mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_metrics] 3 2 2048 SHARED 20 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- ha_mapr-ComputeMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.137 Service Ready ha_mapr-ComputeMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.136 Service Ready ha_mapr-ComputeMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.138 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- Monitor [mapr_zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------- ha_mapr-Monitor-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.152 Service Ready ha_mapr-Monitor-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.149 Service Ready ha_mapr-Monitor-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.148 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 5 2 2048 LOCAL 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-worker-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.145 Service Ready ha_mapr-worker-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.147 Service Ready ha_mapr-worker-4 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.144 Service Ready ha_mapr-worker-3 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready ha_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.143 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 2048 SHARED 30 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-client-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.151 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------- DataMaster [mapr_cldb, mapr_nfs, mapr_fileserver] 3 2 2048 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ ha_mapr-DataMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.135 Service Ready ha_mapr-DataMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.140 Service Ready ha_mapr-DataMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- ha_mapr-mysqlserver-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.150 Service Ready {code} log on client vm and run wordcout job: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvol/input/ /myvol/output/ 13/02/01 01:56:18 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:56:18 INFO input.FileInputFormat: Total input paths to process : 1 13/02/01 01:56:18 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's output directory at /myvol/output 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's user history location directory at /myvol/output/_logs 13/02/01 01:56:18 INFO mapred.JobClient: Running job: job_201301310814_0004 13/02/01 01:56:19 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:56:27 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0004_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:31 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_2, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:43 INFO mapred.JobClient: Job job_201301310814_0004 failed with state FAILED due to: NA 13/02/01 01:56:43 INFO mapred.JobClient: Counters: 8 13/02/01 01:56:43 INFO mapred.JobClient: Job Counters 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=9826 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Rack-local map tasks=1 13/02/01 01:56:43 INFO mapred.JobClient: Launched map tasks=4 13/02/01 01:56:43 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Failed map tasks=1 {code} Benchmark: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar terasort /myvol/terasort-input /myvol/terasort-output 13/02/01 01:53:39 INFO terasort.TeraSort: starting 13/02/01 01:53:39 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:39 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:53:40 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/02/01 01:53:40 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/02/01 01:53:40 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:53:41 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's output directory at maprfs:/myvol/terasort-output 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/myvol/terasort-output/_logs 13/02/01 01:53:41 INFO mapred.JobClient: Running job: job_201301310814_0003 13/02/01 01:53:42 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0003_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0003_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0003_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0003_m_000001_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:54:04 INFO mapred.JobClient: Job job_201301310814_0003 failed with state FAILED due to: NA 13/02/01 01:54:04 INFO mapred.JobClient: Counters: 8 13/02/01 01:54:04 INFO mapred.JobClient: Job Counters 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=15134 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Rack-local map tasks=6 13/02/01 01:54:04 INFO mapred.JobClient: Launched map tasks=8 13/02/01 01:54:04 INFO mapred.JobClient: Data-local map tasks=2 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/02/01 01:51:18 INFO fs.TestDFSIO: nrFiles = 5 13/02/01 01:51:18 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/02/01 01:51:18 INFO fs.TestDFSIO: bufferSize = 1000000 13/02/01 01:51:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/02/01 01:51:19 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 13/02/01 01:51:19 INFO fs.TestDFSIO: created control files for: 5 files 13/02/01 01:51:19 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:51:20 INFO mapred.FileInputFormat: Total input paths to process : 5 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs 13/02/01 01:51:20 INFO mapred.JobClient: Running job: job_201301310814_0001 13/02/01 01:51:21 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:51:30 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_0, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:35 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000002_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_0, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000004_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000003_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_1, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_1, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000002_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_1, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000004_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000003_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:43 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_2, Status : FAILED on node 10.111.88.147 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_2, Status : FAILED on node 10.111.88.145 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_2, Status : FAILED on node 10.111.88.144 Error: Java heap space 13/02/01 01:51:46 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000004_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:47 INFO mapred.JobClient: Job job_201301310814_0001 failed with state FAILED due to: NA 13/02/01 01:51:47 INFO mapred.JobClient: Counters: 8 13/02/01 01:51:47 INFO mapred.JobClient: Job Counters 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=35682 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Rack-local map tasks=16 13/02/01 01:51:47 INFO mapred.JobClient: Launched map tasks=19 13/02/01 01:51:47 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}
    via by Yifeng Xiao,
  • Create a large scale cluster: {code} cluster name: ha_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------------------------------------- ComputeMaster [mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_metrics] 3 2 2048 SHARED 20 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- ha_mapr-ComputeMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.137 Service Ready ha_mapr-ComputeMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.136 Service Ready ha_mapr-ComputeMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.138 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- Monitor [mapr_zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------- ha_mapr-Monitor-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.152 Service Ready ha_mapr-Monitor-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.149 Service Ready ha_mapr-Monitor-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.148 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 5 2 2048 LOCAL 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-worker-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.145 Service Ready ha_mapr-worker-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.147 Service Ready ha_mapr-worker-4 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.144 Service Ready ha_mapr-worker-3 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready ha_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.143 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 2048 SHARED 30 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-client-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.151 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------- DataMaster [mapr_cldb, mapr_nfs, mapr_fileserver] 3 2 2048 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ ha_mapr-DataMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.135 Service Ready ha_mapr-DataMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.140 Service Ready ha_mapr-DataMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- ha_mapr-mysqlserver-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.150 Service Ready {code} log on client vm and run wordcout job: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvol/input/ /myvol/output/ 13/02/01 01:56:18 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:56:18 INFO input.FileInputFormat: Total input paths to process : 1 13/02/01 01:56:18 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's output directory at /myvol/output 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's user history location directory at /myvol/output/_logs 13/02/01 01:56:18 INFO mapred.JobClient: Running job: job_201301310814_0004 13/02/01 01:56:19 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:56:27 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0004_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:31 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_2, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:43 INFO mapred.JobClient: Job job_201301310814_0004 failed with state FAILED due to: NA 13/02/01 01:56:43 INFO mapred.JobClient: Counters: 8 13/02/01 01:56:43 INFO mapred.JobClient: Job Counters 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=9826 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Rack-local map tasks=1 13/02/01 01:56:43 INFO mapred.JobClient: Launched map tasks=4 13/02/01 01:56:43 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Failed map tasks=1 {code} Benchmark: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar terasort /myvol/terasort-input /myvol/terasort-output 13/02/01 01:53:39 INFO terasort.TeraSort: starting 13/02/01 01:53:39 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:39 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:53:40 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/02/01 01:53:40 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/02/01 01:53:40 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:53:41 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's output directory at maprfs:/myvol/terasort-output 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/myvol/terasort-output/_logs 13/02/01 01:53:41 INFO mapred.JobClient: Running job: job_201301310814_0003 13/02/01 01:53:42 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0003_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0003_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0003_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0003_m_000001_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:54:04 INFO mapred.JobClient: Job job_201301310814_0003 failed with state FAILED due to: NA 13/02/01 01:54:04 INFO mapred.JobClient: Counters: 8 13/02/01 01:54:04 INFO mapred.JobClient: Job Counters 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=15134 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Rack-local map tasks=6 13/02/01 01:54:04 INFO mapred.JobClient: Launched map tasks=8 13/02/01 01:54:04 INFO mapred.JobClient: Data-local map tasks=2 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/02/01 01:51:18 INFO fs.TestDFSIO: nrFiles = 5 13/02/01 01:51:18 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/02/01 01:51:18 INFO fs.TestDFSIO: bufferSize = 1000000 13/02/01 01:51:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/02/01 01:51:19 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 13/02/01 01:51:19 INFO fs.TestDFSIO: created control files for: 5 files 13/02/01 01:51:19 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:51:20 INFO mapred.FileInputFormat: Total input paths to process : 5 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs 13/02/01 01:51:20 INFO mapred.JobClient: Running job: job_201301310814_0001 13/02/01 01:51:21 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:51:30 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_0, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:35 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000002_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_0, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000004_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000003_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_1, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_1, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000002_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_1, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000004_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000003_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:43 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_2, Status : FAILED on node 10.111.88.147 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_2, Status : FAILED on node 10.111.88.145 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_2, Status : FAILED on node 10.111.88.144 Error: Java heap space 13/02/01 01:51:46 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000004_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:47 INFO mapred.JobClient: Job job_201301310814_0001 failed with state FAILED due to: NA 13/02/01 01:51:47 INFO mapred.JobClient: Counters: 8 13/02/01 01:51:47 INFO mapred.JobClient: Job Counters 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=35682 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Rack-local map tasks=16 13/02/01 01:51:47 INFO mapred.JobClient: Launched map tasks=19 13/02/01 01:51:47 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}
    via by Yifeng Xiao,
  • Hadoop. Restart Map
    via Stack Overflow by gmlvsv
    ,
  • error with crawling with nutch
    via Stack Overflow by goodi
    ,
    • java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    5 more bugmates