java.io.IOException: Error Reading IndexFile

SpringSource Issue Tracker | Yifeng Xiao | 4 years ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Error Reading IndexFile
  2. 0

    Serengeti server ip: 10.111.89.164 Reproduce steps: 1. Create a cluster with attachment specfile. (Compute node use tempfs) {code} cluster name: sharenfs, distro: apache, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 2048 SHARED 25 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-master-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------- data [tempfs_server, hadoop_datanode] 2 2 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------- sharenfs-data-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.241 Service Ready sharenfs-data-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.116 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------- compute [tempfs_client, hadoop_tasktracker] 4 1 1024 TEMPFS 50 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------- sharenfs-compute-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.139 Service Ready sharenfs-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.135 Service Ready sharenfs-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.136 Service Ready sharenfs-compute-3 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.130 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- client [hadoop_client] 1 1 3748 SHARED 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- sharenfs-client-0 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.76 Service Ready {code} 2. Run terasort failed in client vm: {code} [joe@10 ~]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output 13/03/12 09:04:08 INFO terasort.TeraSort: starting 13/03/12 09:04:09 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/03/12 09:04:10 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/03/12 09:04:10 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/03/12 09:04:11 INFO mapred.FileInputFormat: Total input paths to process : 2 13/03/12 09:04:11 INFO mapred.JobClient: Running job: job_201303120619_0009 13/03/12 09:04:12 INFO mapred.JobClient: map 0% reduce 0% 13/03/12 09:04:29 INFO mapred.JobClient: map 12% reduce 0% 13/03/12 09:04:32 INFO mapred.JobClient: map 37% reduce 0% 13/03/12 09:04:33 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:35 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:37 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:40 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_0, Status : FAILED 13/03/12 09:04:41 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:42 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:43 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000007_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000007_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:04:44 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:46 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_0, Status : FAILED 13/03/12 09:04:47 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:04:49 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:51 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_0, Status : FAILED 13/03/12 09:04:54 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:56 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_0, Status : FAILED 13/03/12 09:04:57 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:04:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_0, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000006_0,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:00 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:01 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_0, Status : FAILED 13/03/12 09:05:02 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:05 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:06 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000003_0, Status : FAILED 13/03/12 09:05:07 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:09 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:11 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_1, Status : FAILED 13/03/12 09:05:12 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:14 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:16 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_1, Status : FAILED 13/03/12 09:05:19 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:21 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_1, Status : FAILED 13/03/12 09:05:22 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:25 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:27 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000005_1, Status : FAILED 13/03/12 09:05:28 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:29 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:31 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_1, Status : FAILED 13/03/12 09:05:32 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:33 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000001_2, Status : FAILED Map output lost, rescheduling: getMapOutput(attempt_201303120619_0009_m_000001_2,0) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000001_2/output/file.out.index did not match expected owner 'root' at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177) at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more 13/03/12 09:05:35 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:36 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000000_1, Status : FAILED 13/03/12 09:05:37 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:39 INFO mapred.JobClient: map 62% reduce 0% 13/03/12 09:05:40 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:44 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000004_2, Status : FAILED 13/03/12 09:05:45 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:05:49 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000006_1, Status : FAILED 13/03/12 09:05:50 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:53 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:54 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000007_2, Status : FAILED 13/03/12 09:05:55 INFO mapred.JobClient: map 75% reduce 0% 13/03/12 09:05:57 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:05:59 INFO mapred.JobClient: Task Id : attempt_201303120619_0009_m_000002_2, Status : FAILED 13/03/12 09:06:01 INFO mapred.JobClient: map 100% reduce 0% 13/03/12 09:06:02 INFO mapred.JobClient: map 87% reduce 0% 13/03/12 09:06:05 INFO mapred.JobClient: Job complete: job_201303120619_0009 13/03/12 09:06:05 INFO mapred.JobClient: Counters: 25 13/03/12 09:06:05 INFO mapred.JobClient: Job Counters 13/03/12 09:06:05 INFO mapred.JobClient: Launched reduce tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=291487 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/03/12 09:06:05 INFO mapred.JobClient: Rack-local map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: Launched map tasks=26 13/03/12 09:06:05 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=89541 13/03/12 09:06:05 INFO mapred.JobClient: Failed map tasks=1 13/03/12 09:06:05 INFO mapred.JobClient: File Input Format Counters 13/03/12 09:06:05 INFO mapred.JobClient: Bytes Read=365798660 13/03/12 09:06:05 INFO mapred.JobClient: FileSystemCounters 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_READ=774 13/03/12 09:06:05 INFO mapred.JobClient: HDFS_BYTES_READ=365799332 13/03/12 09:06:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=373236984 13/03/12 09:06:05 INFO mapred.JobClient: Map-Reduce Framework 13/03/12 09:06:05 INFO mapred.JobClient: Map output materialized bytes=373097982 13/03/12 09:06:05 INFO mapred.JobClient: Map input records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Spilled Records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Map output bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: Total committed heap usage (bytes)=1198026752 13/03/12 09:06:05 INFO mapred.JobClient: CPU time spent (ms)=21930 13/03/12 09:06:05 INFO mapred.JobClient: Map input bytes=365782300 13/03/12 09:06:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=672 13/03/12 09:06:05 INFO mapred.JobClient: Combine input records=0 13/03/12 09:06:05 INFO mapred.JobClient: Combine output records=0 13/03/12 09:06:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=1080045568 13/03/12 09:06:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2786127872 13/03/12 09:06:05 INFO mapred.JobClient: Map output records=3657823 13/03/12 09:06:05 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303120619_0009_m_000004 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) {code} Terasort 5M data: {code} hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teragen 5000000 /user/joe/terasort-input hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar terasort /user/joe/terasort-input /user/joe/terasort-output hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.1.jar teravalidate /user/joe/terasort-output /user/joe/terasort {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Error Reading IndexFile
  3. 0

    Hadoop job failure : Error Reading IndexFile?

    Stack Overflow | 5 years ago | daydreamer
    java.io.IOException: Owner '4294967294' for path /work/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201204251959_0001/attempt_201204251959_0001_m_000000_0/output/file.out.index did not match expected owner 'hduser'
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hadoop MR jobs failed: Owner '<uid>' for path */jobcache/job_*/attempt_*/output/file.out.index did not match expected owner '<username>'

    Google Groups | 5 years ago | Dirk Meister
    java.io.IOException: Owner 'MYUID' for path LOCAL_PATH/jobcache/job_201202210928_0005/attempt_201202210928_0005_m_000008_0/output/file.out did not match expected owner 'MYUSERNAME'

    Root Cause Analysis

    1. java.io.IOException

      Owner 'mapred' for path /mnt/mapred/export0/sharenfs-compute-3/taskTracker/joe/jobcache/job_201303120619_0009/attempt_201303120619_0009_m_000006_0/output/file.out.index did not match expected owner 'root'

      at org.apache.hadoop.io.SecureIOUtils.checkStat()
    2. Hadoop
      SecureIOUtils.openForRead
      1. org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:177)
      2. org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:117)
      2 frames
    3. Hadoop
      TaskTracker$MapOutputServlet.doGet
      1. org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:61)
      2. org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:54)
      3. org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109)
      4. org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66)
      5. org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3850)
      5 frames
    4. JavaServlet
      HttpServlet.service
      1. javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      2. javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      2 frames
    5. Jetty Server
      ServletHandler$CachedChain.doFilter
      1. org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      2. org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      2 frames
    6. Hadoop
      HttpServer$QuotingInputFilter.doFilter
      1. org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
      1 frame
    7. Jetty Server
      SelectChannelEndPoint.run
      1. org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      2. org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      3. org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      4. org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      5. org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      6. org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      7. org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      8. org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      9. org.mortbay.jetty.Server.handle(Server.java:326)
      10. org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      11. org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      12. org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      13. org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      14. org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      15. org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      15 frames
    8. Jetty Util
      QueuedThreadPool$PoolThread.run
      1. org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      1 frame