java.io.IOException: Error Reading IndexFile

SpringSource Issue Tracker | Yifeng Xiao | 4 years ago
  1. 0

    Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Error Reading IndexFile
  2. 0

    Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Error Reading IndexFile
  3. 0

    Hadoop job failure : Error Reading IndexFile?

    Stack Overflow | 5 years ago | daydreamer
    java.io.IOException: Owner '4294967294' for path /work/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201204251959_0001/attempt_201204251959_0001_m_000000_0/output/file.out.index did not match expected owner 'hduser'
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hadoop MR jobs failed: Owner '<uid>' for path */jobcache/job_*/attempt_*/output/file.out.index did not match expected owner '<username>'

    Google Groups | 5 years ago | Dirk Meister
    java.io.IOException: Owner 'MYUID' for path LOCAL_PATH/jobcache/job_201202210928_0005/attempt_201202210928_0005_m_000008_0/output/file.out did not match expected owner 'MYUSERNAME'
  6. 0

    How to read Hadoop Map intermediate file file.out

    Stack Overflow | 3 years ago | Prashant
    java.io.IOException: /opt/hadoopws/tmp/mapred/local/taskTracker/hduser/jobcache/job_201403260903_0001/attempt_201403260903_0001_m_000000_0/output/file.out not a SequenceFile

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root'

      at org.apache.hadoop.io.SecureIOUtils.checkStat()
    2. Hadoop
      SecureIOUtils.openForRead
      1. org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177)
      2. org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117)
      2 frames
    3. Hadoop
      TaskTracker$MapOutputServlet.doGet
      1. org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61)
      2. org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54)
      3. org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109)
      4. org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66)
      5. org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850)
      5 frames
    4. JavaServlet
      HttpServlet.service
      1. javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      2. javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      2 frames
    5. Jetty Server
      ServletHandler$CachedChain.doFilter
      1. org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      2. org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      2 frames
    6. Hadoop
      HttpServer$QuotingInputFilter.doFilter
      1. org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
      1 frame
    7. Jetty Server
      SelectChannelEndPoint.run
      1. org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      2. org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      3. org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      4. org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      5. org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      6. org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      7. org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      8. org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      9. org.mortbay.jetty.Server.handle(Server.java:326)
      10. org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      11. org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      12. org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      13. org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      14. org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      15. org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      15 frames
    8. Jetty Util
      QueuedThreadPool$PoolThread.run
      1. org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      1 frame