java.io.IOException: Job failed!

SpringSource Issue Tracker | Yifeng Xiao | 4 years ago
  1. 0

    Create a large scale cluster: {code} cluster name: ha_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------------------------------------- ComputeMaster [mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_metrics] 3 2 2048 SHARED 20 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- ha_mapr-ComputeMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.137 Service Ready ha_mapr-ComputeMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.136 Service Ready ha_mapr-ComputeMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.138 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- Monitor [mapr_zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------- ha_mapr-Monitor-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.152 Service Ready ha_mapr-Monitor-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.149 Service Ready ha_mapr-Monitor-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.148 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 5 2 2048 LOCAL 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-worker-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.145 Service Ready ha_mapr-worker-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.147 Service Ready ha_mapr-worker-4 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.144 Service Ready ha_mapr-worker-3 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready ha_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.143 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 2048 SHARED 30 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-client-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.151 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------- DataMaster [mapr_cldb, mapr_nfs, mapr_fileserver] 3 2 2048 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ ha_mapr-DataMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.135 Service Ready ha_mapr-DataMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.140 Service Ready ha_mapr-DataMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- ha_mapr-mysqlserver-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.150 Service Ready {code} log on client vm and run wordcout job: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvol/input/ /myvol/output/ 13/02/01 01:56:18 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:56:18 INFO input.FileInputFormat: Total input paths to process : 1 13/02/01 01:56:18 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's output directory at /myvol/output 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's user history location directory at /myvol/output/_logs 13/02/01 01:56:18 INFO mapred.JobClient: Running job: job_201301310814_0004 13/02/01 01:56:19 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:56:27 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0004_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:31 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_2, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:43 INFO mapred.JobClient: Job job_201301310814_0004 failed with state FAILED due to: NA 13/02/01 01:56:43 INFO mapred.JobClient: Counters: 8 13/02/01 01:56:43 INFO mapred.JobClient: Job Counters 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=9826 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Rack-local map tasks=1 13/02/01 01:56:43 INFO mapred.JobClient: Launched map tasks=4 13/02/01 01:56:43 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Failed map tasks=1 {code} Benchmark: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar terasort /myvol/terasort-input /myvol/terasort-output 13/02/01 01:53:39 INFO terasort.TeraSort: starting 13/02/01 01:53:39 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:39 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:53:40 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/02/01 01:53:40 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/02/01 01:53:40 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:53:41 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's output directory at maprfs:/myvol/terasort-output 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/myvol/terasort-output/_logs 13/02/01 01:53:41 INFO mapred.JobClient: Running job: job_201301310814_0003 13/02/01 01:53:42 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0003_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0003_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0003_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0003_m_000001_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:54:04 INFO mapred.JobClient: Job job_201301310814_0003 failed with state FAILED due to: NA 13/02/01 01:54:04 INFO mapred.JobClient: Counters: 8 13/02/01 01:54:04 INFO mapred.JobClient: Job Counters 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=15134 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Rack-local map tasks=6 13/02/01 01:54:04 INFO mapred.JobClient: Launched map tasks=8 13/02/01 01:54:04 INFO mapred.JobClient: Data-local map tasks=2 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/02/01 01:51:18 INFO fs.TestDFSIO: nrFiles = 5 13/02/01 01:51:18 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/02/01 01:51:18 INFO fs.TestDFSIO: bufferSize = 1000000 13/02/01 01:51:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/02/01 01:51:19 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 13/02/01 01:51:19 INFO fs.TestDFSIO: created control files for: 5 files 13/02/01 01:51:19 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:51:20 INFO mapred.FileInputFormat: Total input paths to process : 5 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs 13/02/01 01:51:20 INFO mapred.JobClient: Running job: job_201301310814_0001 13/02/01 01:51:21 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:51:30 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_0, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:35 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000002_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_0, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000004_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000003_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_1, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_1, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000002_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_1, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000004_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000003_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:43 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_2, Status : FAILED on node 10.111.88.147 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_2, Status : FAILED on node 10.111.88.145 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_2, Status : FAILED on node 10.111.88.144 Error: Java heap space 13/02/01 01:51:46 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000004_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:47 INFO mapred.JobClient: Job job_201301310814_0001 failed with state FAILED due to: NA 13/02/01 01:51:47 INFO mapred.JobClient: Counters: 8 13/02/01 01:51:47 INFO mapred.JobClient: Job Counters 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=35682 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Rack-local map tasks=16 13/02/01 01:51:47 INFO mapred.JobClient: Launched map tasks=19 13/02/01 01:51:47 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Job failed!
  2. 0

    Create a large scale cluster: {code} cluster name: ha_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------------------------------------- ComputeMaster [mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_metrics] 3 2 2048 SHARED 20 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- ha_mapr-ComputeMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.137 Service Ready ha_mapr-ComputeMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.136 Service Ready ha_mapr-ComputeMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.138 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- Monitor [mapr_zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------- ha_mapr-Monitor-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.152 Service Ready ha_mapr-Monitor-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.149 Service Ready ha_mapr-Monitor-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.148 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 5 2 2048 LOCAL 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-worker-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.145 Service Ready ha_mapr-worker-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.147 Service Ready ha_mapr-worker-4 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.144 Service Ready ha_mapr-worker-3 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready ha_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.143 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 2048 SHARED 30 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------- ha_mapr-client-0 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.151 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------- DataMaster [mapr_cldb, mapr_nfs, mapr_fileserver] 3 2 2048 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ ha_mapr-DataMaster-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.135 Service Ready ha_mapr-DataMaster-1 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.140 Service Ready ha_mapr-DataMaster-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- ha_mapr-mysqlserver-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.150 Service Ready {code} log on client vm and run wordcout job: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvol/input/ /myvol/output/ 13/02/01 01:56:18 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:56:18 INFO input.FileInputFormat: Total input paths to process : 1 13/02/01 01:56:18 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's output directory at /myvol/output 13/02/01 01:56:18 INFO mapred.JobClient: Creating job's user history location directory at /myvol/output/_logs 13/02/01 01:56:18 INFO mapred.JobClient: Running job: job_201301310814_0004 13/02/01 01:56:19 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:56:27 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0004_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:31 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0004_m_000000_2, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:56:43 INFO mapred.JobClient: Job job_201301310814_0004 failed with state FAILED due to: NA 13/02/01 01:56:43 INFO mapred.JobClient: Counters: 8 13/02/01 01:56:43 INFO mapred.JobClient: Job Counters 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=9826 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Rack-local map tasks=1 13/02/01 01:56:43 INFO mapred.JobClient: Launched map tasks=4 13/02/01 01:56:43 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:56:43 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:56:43 INFO mapred.JobClient: Failed map tasks=1 {code} Benchmark: {code} [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar terasort /myvol/terasort-input /myvol/terasort-output 13/02/01 01:53:39 INFO terasort.TeraSort: starting 13/02/01 01:53:39 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:39 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/01 01:53:40 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 13/02/01 01:53:40 INFO compress.CodecPool: Got brand-new compressor Making 1 from 100000 records Step size is 100000.0 13/02/01 01:53:40 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:53:41 INFO mapred.FileInputFormat: Total input paths to process : 2 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's output directory at maprfs:/myvol/terasort-output 13/02/01 01:53:41 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/myvol/terasort-output/_logs 13/02/01 01:53:41 INFO mapred.JobClient: Running job: job_201301310814_0003 13/02/01 01:53:42 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0003_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:50 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0003_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:55 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0003_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000001_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0003_m_000001_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000001_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:53:58 INFO mapred.JobClient: Task Id : attempt_201301310814_0003_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0003_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0003_m_000000_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:54:04 INFO mapred.JobClient: Job job_201301310814_0003 failed with state FAILED due to: NA 13/02/01 01:54:04 INFO mapred.JobClient: Counters: 8 13/02/01 01:54:04 INFO mapred.JobClient: Job Counters 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=15134 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Rack-local map tasks=6 13/02/01 01:54:04 INFO mapred.JobClient: Launched map tasks=8 13/02/01 01:54:04 INFO mapred.JobClient: Data-local map tasks=2 13/02/01 01:54:04 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:54:04 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) [mapr@10 ~]$ hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/02/01 01:51:18 INFO fs.TestDFSIO: nrFiles = 5 13/02/01 01:51:18 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/02/01 01:51:18 INFO fs.TestDFSIO: bufferSize = 1000000 13/02/01 01:51:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/02/01 01:51:19 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 13/02/01 01:51:19 INFO fs.TestDFSIO: created control files for: 5 files 13/02/01 01:51:19 INFO fs.JobTrackerWatcher: Current running JobTracker is: /10.111.88.138:9001 13/02/01 01:51:20 INFO mapred.FileInputFormat: Total input paths to process : 5 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write 13/02/01 01:51:20 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs 13/02/01 01:51:20 INFO mapred.JobClient: Running job: job_201301310814_0001 13/02/01 01:51:21 INFO mapred.JobClient: map 0% reduce 0% 13/02/01 01:51:30 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_0, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000000_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:35 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_0, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000002_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:36 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_0, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000001_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_0, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000004_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:37 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_0, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000003_0: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_0: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_1, Status : FAILED on node 10.111.88.144 Error: Java heap space attempt_201301310814_0001_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000000_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:39 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_1, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000002_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000002_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_1, Status : FAILED on node 10.111.88.146 Error: Java heap space attempt_201301310814_0001_m_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000001_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_1, Status : FAILED on node 10.111.88.147 Error: Java heap space attempt_201301310814_0001_m_000004_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:41 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_1, Status : FAILED on node 10.111.88.145 Error: Java heap space attempt_201301310814_0001_m_000003_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000003_1: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:43 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000001_2, Status : FAILED on node 10.111.88.147 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000000_2, Status : FAILED on node 10.111.88.146 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000002_2, Status : FAILED on node 10.111.88.145 Error: Java heap space 13/02/01 01:51:45 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000003_2, Status : FAILED on node 10.111.88.144 Error: Java heap space 13/02/01 01:51:46 INFO mapred.JobClient: Task Id : attempt_201301310814_0001_m_000004_2, Status : FAILED on node 10.111.88.143 Error: Java heap space attempt_201301310814_0001_m_000004_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). attempt_201301310814_0001_m_000004_2: log4j:WARN Please initialize the log4j system properly. 13/02/01 01:51:47 INFO mapred.JobClient: Job job_201301310814_0001 failed with state FAILED due to: NA 13/02/01 01:51:47 INFO mapred.JobClient: Counters: 8 13/02/01 01:51:47 INFO mapred.JobClient: Job Counters 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of mappers(ms)=35682 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Rack-local map tasks=16 13/02/01 01:51:47 INFO mapred.JobClient: Launched map tasks=19 13/02/01 01:51:47 INFO mapred.JobClient: Data-local map tasks=3 13/02/01 01:51:47 INFO mapred.JobClient: Aggregate execution time of reducers(ms)=0 13/02/01 01:51:47 INFO mapred.JobClient: Failed map tasks=1 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Job failed!
  3. 0

    TestDFSIO fails to run

    Google Groups | 6 years ago | Da Zheng
    java.io.IOException: Job failed!
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Simple YARN benchmark TestDFSIO fails

    Stack Overflow | 3 years ago | toom
    java.io.IOException: Job failed!
  6. 0

    TestDFSIO with Tachyon

    Google Groups | 2 years ago | Kerkinos Ioannis
    java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

    9 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Job failed!

      at org.apache.hadoop.mapred.JobClient.runJob()
    2. Hadoop
      JobClient.runJob
      1. org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1334)
      1 frame
    3. Hadoop
      TestDFSIO.main
      1. org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:302)
      2. org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:281)
      3. org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:680)
      4. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      5. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      6. org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537)
      6 frames
    4. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    5. Hadoop
      ProgramDriver.driver
      1. org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      2. org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      2 frames
    6. org.apache.hadoop
      AllTestDriver.main
      1. org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83)
      1 frame
    7. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    8. Hadoop
      RunJar.main
      1. org.apache.hadoop.util.RunJar.main(RunJar.java:197)
      1 frame