org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

Apache's JIRA Issue Tracker | Takahiko Saito | 1 year ago
tip
Do you find the tips below useful? Click on the to mark them and say thanks to Rafael Fischer Viola and rafael . Or join the community to write better ones.
  1. 0
    samebug tip
    See if you placed serde.jar in the correct folder on all the node-hosts
  2. 0
    samebug tip
    Return 2 is sometimes a hadoop/yarn memory problem! If you're running a single-node cluster, check this https://goo.gl/C1jkw8
  3. 0

    The following query fails, when Azure Filesystem is used as default file system, and HDFS is used for intermediate data. {noformat} >>> create temporary table s10k stored as orc as select * from studenttab10k; >>> create temporary table v10k as select * from votertab10k; >>> select registration from s10k s join v10k v on (s.name = v.name) join studentparttab30k p on (p.name = v.name) where s.age < 25 and v.age < 25 and p.age < 25; ERROR : Execution failed with exit status: 2 ERROR : Obtaining error information ERROR : Task failed! Task ID: Stage-5 Logs: ERROR : /var/log/hive/hiveServer2.log Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2) Aborting command set because "force" is false and command failed: "select registration from s10k s join v10k v on (s.name = v.name) join studentparttab30k p on (p.name = v.name) where s.age < 25 and v.age < 25 and p.age < 25;" Closing: 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@HDINSIGHT.NET;transportMode=http;httpPath=cliservice hiveServer2.log shows: 2016-02-02 18:04:34,182 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,199 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,212 INFO [HiveServer2-HttpHandler-Pool: Thread-55]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(127)) - Could not validate cookie sent, will try to generate a new cookie 2016-02-02 18:04:34,213 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (Driver.java:checkConcurrency(168)) - Concurrency mode is disabled, not creating a lock manager 2016-02-02 18:04:34,219 INFO [HiveServer2-HttpHandler-Pool: Thread-55]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(352)) - Failed to authenticate with http/_HOST kerberos principal, trying with hive/_HOST kerberos principal 2016-02-02 18:04:34,219 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,225 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (Driver.java:execute(1390)) - Setting caller context to query id hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0 2016-02-02 18:04:34,226 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (Driver.java:execute(1393)) - Starting command(queryId=hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0): select registration from s10k s join v10k v on (s.name = v.name) join studentparttab30k p on (p.name = v.name) where s.age < 25 and v.age < 25 and p.age < 25 2016-02-02 18:04:34,228 INFO [HiveServer2-Background-Pool: Thread-517]: hooks.ATSHook (ATSHook.java:<init>(90)) - Created ATS Hook 2016-02-02 18:04:34,229 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,237 INFO [HiveServer2-HttpHandler-Pool: Thread-55]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(169)) - Cookie added for clientUserName hrt_qa 2016-02-02 18:04:34,238 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1454436274229 end=1454436274238 duration=9 from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,239 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,240 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook start=1454436274239 end=1454436274240 duration=1 from=org.apache.hadoop.hive.ql.Driver> Query ID = hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0 2016-02-02 18:04:34,242 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (SessionState.java:printInfo(923)) - Query ID = hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0 Total jobs = 1 2016-02-02 18:04:34,243 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (SessionState.java:printInfo(923)) - Total jobs = 1 2016-02-02 18:04:34,245 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=TimeToSubmit start=1454436274199 end=1454436274245 duration=46 from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,246 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,247 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=task.MAPREDLOCAL.Stage-5 from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:34,258 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (Driver.java:launchTask(1718)) - Starting task [Stage-5:MAPREDLOCAL] in serial mode 2016-02-02 18:04:34,280 INFO [HiveServer2-Background-Pool: Thread-517]: mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(158)) - Generating plan file file:/tmp/hive/916e3dbb-a10d-4888-a063-52fb058ea421/hive_2016-02-02_18-04-29_153_625340340820843828-4/-local-10006/plan.xml 2016-02-02 18:04:34,288 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities> 2016-02-02 18:04:34,289 INFO [HiveServer2-Background-Pool: Thread-517]: exec.Utilities (Utilities.java:serializePlan(1028)) - Serializing MapredLocalWork via kryo 2016-02-02 18:04:34,290 INFO [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPreHookEvent(158)) - Received pre-hook notification for :hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0 2016-02-02 18:04:34,358 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=serializePlan start=1454436274288 end=1454436274358 duration=70 from=org.apache.hadoop.hive.ql.exec.Utilities> 2016-02-02 18:04:34,737 INFO [HiveServer2-Background-Pool: Thread-517]: mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(287)) - Executing: /usr/hdp/2.4.1.0-170/hadoop/bin/hadoop jar /usr/hdp/2.4.1.0-170/hive/lib/hive-common-1.2.1000.2.4.1.0-170.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan file:/tmp/hive/916e3dbb-a10d-4888-a063-52fb058ea421/hive_2016-02-02_18-04-29_153_625340340820843828-4/-local-10006/plan.xml -jobconffile file:/tmp/hive/916e3dbb-a10d-4888-a063-52fb058ea421/hive_2016-02-02_18-04-29_153_625340340820843828-4/-local-10007/jobconf.xml WARNING: Use "yarn jar" to launch YARN applications. 2016-02-02 18:04:37,450 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0]: provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(312)) - Audit Status Log: name=hiveServer2.async.summary.batch.solr, interval=01:21.012 minutes, events=2, succcessCount=2, totalEvents=4, totalSuccessCount=4 Execution log at: /tmp/hive/hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0.log 2016-02-02 18:04:39,248 INFO [HiveServer2-HttpHandler-Pool: Thread-55]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(127)) - Could not validate cookie sent, will try to generate a new cookie 2016-02-02 18:04:39,254 INFO [HiveServer2-HttpHandler-Pool: Thread-55]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(352)) - Failed to authenticate with http/_HOST kerberos principal, trying with hive/_HOST kerberos principal 2016-02-02 18:04:39,261 INFO [HiveServer2-HttpHandler-Pool: Thread-55]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(169)) - Cookie added for clientUserName hrt_qa 2016-02-02 18:04:40 Starting to launch local task to process map join; maximum memory = 477102080 Execution failed with exit status: 2 2016-02-02 18:04:43,728 ERROR [HiveServer2-Background-Pool: Thread-517]: exec.Task (SessionState.java:printError(932)) - Execution failed with exit status: 2 Obtaining error information 2016-02-02 18:04:43,730 ERROR [HiveServer2-Background-Pool: Thread-517]: exec.Task (SessionState.java:printError(932)) - Obtaining error information Task failed! Task ID: Stage-5 Logs: 2016-02-02 18:04:43,730 ERROR [HiveServer2-Background-Pool: Thread-517]: exec.Task (SessionState.java:printError(932)) - Task failed! Task ID: Stage-5 Logs: /var/log/hive/hiveServer2.log 2016-02-02 18:04:43,732 ERROR [HiveServer2-Background-Pool: Thread-517]: exec.Task (SessionState.java:printError(932)) - /var/log/hive/hiveServer2.log 2016-02-02 18:04:43,732 ERROR [HiveServer2-Background-Pool: Thread-517]: mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(307)) - Execution failed with exit status: 2 2016-02-02 18:04:43,733 INFO [HiveServer2-Background-Pool: Thread-517]: hooks.ATSHook (ATSHook.java:<init>(90)) - Created ATS Hook 2016-02-02 18:04:43,734 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:43,736 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1454436283734 end=1454436283736 duration=2 from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:43,736 INFO [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(193)) - Received post-hook notification for :hive_20160202180429_7ffff6ab-64d6-4c89-88b0-6355cc5acbd0 FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 2016-02-02 18:04:43,757 ERROR [HiveServer2-Background-Pool: Thread-517]: ql.Driver (SessionState.java:printError(932)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 2016-02-02 18:04:43,758 INFO [HiveServer2-Background-Pool: Thread-517]: ql.Driver (Driver.java:execute(1621)) - Resetting the caller context to HIVE_SSN_ID:916e3dbb-a10d-4888-a063-52fb058ea421 2016-02-02 18:04:43,759 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=Driver.execute start=1454436274219 end=1454436283759 duration=9540 from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:43,760 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:43,761 INFO [HiveServer2-Background-Pool: Thread-517]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=releaseLocks start=1454436283760 end=1454436283761 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2016-02-02 18:04:43,766 ERROR [HiveServer2-Background-Pool: Thread-517]: operation.Operation (SQLOperation.java:run(209)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) hive Configs can be viewed from http://qelog.hortonworks.com/log/hs21-hbs24-1454382123/artifacts/tmpModifyConfDir_1454394513592/ Attachments Drop files to attach, or browse. Add LinkIssue Links relates to Bug - A problem which impairs or prevents the functions of the product. HIVE-739 webhcat tests failing in HDInsight secure cluster throwing NullPointerException Blocker - Blocks development and/or testing work, production could not run. RESOLVED Activity All Comments Work Log History Activity Ascending order - Click to sort in descending order Permalink Edit Delete tsaito Takahiko Saito added a comment - 3 days ago The test passes via hive CLI and explain shows: 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> explain select registration 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> from s10k s join v10k v 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> on (s.name = v.name) join studentparttab30k p 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> on (p.name = v.name) 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> where s.age < 25 and v.age < 25 and p.age < 25; +--------------------------------------------------------------------------------------------------------------+--+ | Explain | +--------------------------------------------------------------------------------------------------------------+--+ | STAGE DEPENDENCIES: | | Stage-5 is a root stage | | Stage-4 depends on stages: Stage-5 | | Stage-0 depends on stages: Stage-4 | | | | STAGE PLANS: | | Stage: Stage-5 | | Map Reduce Local Work | | Alias -> Map Local Tables: | | s | | Fetch Operator | | limit: -1 | | v | | Fetch Operator | | limit: -1 | | Alias -> Map Local Operator Tree: | | s | | TableScan | | alias: s | | filterExpr: (name is not null and (age < 25)) (type: boolean) | | Statistics: Num rows: 459 Data size: 47777 Basic stats: COMPLETE Column stats: NONE | | Filter Operator | | predicate: (name is not null and (age < 25)) (type: boolean) | | Statistics: Num rows: 76 Data size: 7910 Basic stats: COMPLETE Column stats: NONE | | HashTable Sink Operator | | keys: | | 0 name (type: string) | | 1 name (type: string) | | 2 name (type: string) | | v | | TableScan | | alias: v | | filterExpr: (name is not null and (age < 25)) (type: boolean) | | Statistics: Num rows: 1653 Data size: 337233 Basic stats: COMPLETE Column stats: NONE | | Filter Operator | | predicate: (name is not null and (age < 25)) (type: boolean) | | Statistics: Num rows: 275 Data size: 56103 Basic stats: COMPLETE Column stats: NONE | | HashTable Sink Operator | | keys: | | 0 name (type: string) | | 1 name (type: string) | | 2 name (type: string) | | | | Stage: Stage-4 | | Map Reduce | | Map Operator Tree: | | TableScan | | alias: p | | filterExpr: (name is not null and (age < 25)) (type: boolean) | | Statistics: Num rows: 30000 Data size: 627520 Basic stats: COMPLETE Column stats: COMPLETE | | Filter Operator | | predicate: (name is not null and (age < 25)) (type: boolean) | | Statistics: Num rows: 10000 Data size: 1010000 Basic stats: COMPLETE Column stats: COMPLETE | | Map Join Operator | | condition map: | | Inner Join 0 to 1 | | Inner Join 1 to 2 | | keys: | | 0 name (type: string) | | 1 name (type: string) | | 2 name (type: string) | | outputColumnNames: _col8 | | Statistics: Num rows: 22000 Data size: 2222000 Basic stats: COMPLETE Column stats: NONE | | Select Operator | | expressions: _col8 (type: string) | | outputColumnNames: _col0 | | Statistics: Num rows: 22000 Data size: 2222000 Basic stats: COMPLETE Column stats: NONE | | File Output Operator | | compressed: false | | Statistics: Num rows: 22000 Data size: 2222000 Basic stats: COMPLETE Column stats: NONE | | table: | | input format: org.apache.hadoop.mapred.TextInputFormat | | output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | | serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | | Local Work: | | Map Reduce Local Work | | | | Stage: Stage-0 | | Fetch Operator | | limit: -1 | | Processor Tree: | | ListSink | | | +--------------------------------------------------------------------------------------------------------------+--+ 83 rows selected (2.473 seconds) Permalink Edit Delete tsaito Takahiko Saito added a comment - 3 days ago emptablemisc_8 also fails with the same error: >>> create temporary table temp1 as select * from votertab10k; >>> select * from studenttab10k s where s.name not in (select name from temp1); INFO : Number of reduce tasks determined at compile time: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:1 INFO : Submitting tokens for job: job_1454394534358_0164 INFO : Kind: HDFS_DELEGATION_TOKEN, Service: 10.0.0.36:8020, Ident: (HDFS_DELEGATION_TOKEN token 246 for hrt_qa) INFO : The url to track the job: http://hn0-hs21-h.hdinsight.net:8088/proxy/application_1454394534358_0164/ INFO : Starting Job = job_1454394534358_0164, Tracking URL = http://hn0-hs21-h.hdinsight.net:8088/proxy/application_1454394534358_0164/ INFO : Kill Command = /usr/hdp/2.4.1.0-170/hadoop/bin/hadoop job -kill job_1454394534358_0164 INFO : Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 1 INFO : 2016-02-02 10:11:02,367 Stage-4 map = 0%, reduce = 0% INFO : 2016-02-02 10:11:26,060 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 5.48 sec INFO : 2016-02-02 10:11:39,024 Stage-4 map = 100%, reduce = 100%, Cumulative CPU 11.18 sec INFO : MapReduce Total cumulative CPU time: 11 seconds 180 msec INFO : Ended Job = job_1454394534358_0164 INFO : Stage-9 is selected by condition resolver. INFO : Stage-1 is filtered out by condition resolver. ERROR : Execution failed with exit status: 2 ERROR : Obtaining error information ERROR : Task failed! Task ID: Stage-9 Logs: ERROR : /var/log/hive/hiveServer2.log INFO : Number of reduce tasks determined at compile time: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:2 INFO : Submitting tokens for job: job_1454394534358_0169 INFO : Kind: HDFS_DELEGATION_TOKEN, Service: 10.0.0.36:8020, Ident: (HDFS_DELEGATION_TOKEN token 252 for hrt_qa) INFO : The url to track the job: http://hn0-hs21-h.hdinsight.net:8088/proxy/application_1454394534358_0169/ INFO : Starting Job = job_1454394534358_0169, Tracking URL = http://hn0-hs21-h.hdinsight.net:8088/proxy/application_1454394534358_0169/ INFO : Kill Command = /usr/hdp/2.4.1.0-170/hadoop/bin/hadoop job -kill job_1454394534358_0169 INFO : Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1 INFO : 2016-02-02 10:16:38,027 Stage-1 map = 0%, reduce = 0% INFO : 2016-02-02 10:16:52,498 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 3.8 sec INFO : 2016-02-02 10:16:53,566 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 9.94 sec INFO : 2016-02-02 10:17:16,202 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 17.04 sec INFO : MapReduce Total cumulative CPU time: 17 seconds 40 msec INFO : Ended Job = job_1454394534358_0169 ERROR : Execution failed with exit status: 2 ERROR : Obtaining error information ERROR : Task failed! Task ID: Stage-8 Logs: ERROR : /var/log/hive/hiveServer2.log Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2) Aborting command set because "force" is false and command failed: "select * from studenttab10k s where s.name not in (select name from temp1);" Closing: 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@HDINSIGHT.NET;transportMode=http;httpPath=cliservice Its app log can be viewed at http://qelog.hortonworks.com/log/hs21-hbs24-1454382123/app-logs/application_1454394534358_0169.log Permalink Edit Delete tsaito Takahiko Saito added a comment - 3 days ago The query works without 'temporary table': 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> select registration from studenttab10k s join votertab10k v 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> on (s.name = v.name) join studentparttab30k p 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> on (p.name = v.name) 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> where s.age < 25 and v.age < 25 and p.age < 25; Permalink Edit Delete tsaito Takahiko Saito added a comment - 3 days ago More info about temporary table: 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> describe formatted s10k; +-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--+ | col_name | data_type | comment | +-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--+ | # col_name | data_type | comment | | | NULL | NULL | | name | string | | | age | int | | | gpa | double | | | | NULL | NULL | | # Detailed Table Information | NULL | NULL | | Database: | default | NULL | | Owner: | hrt_qa | NULL | | CreateTime: | Tue Feb 02 23:02:31 UTC 2016 | NULL | | LastAccessTime: | UNKNOWN | NULL | | Protect Mode: | None | NULL | | Retention: | 0 | NULL | | Location: | hdfs://hn0-hs21-h.hdinsight.net:8020/tmp/hive/hive/de2667ea-5bc1-4548-a4c4-97e46d6081f8/_tmp_space.db/8e8482c9-a2a4-4e8c-ad0f-2cd6e8fcdb4c | NULL | | Table Type: | MANAGED_TABLE | NULL | | | NULL | NULL | | # Storage Information | NULL | NULL | | SerDe Library: | org.apache.hadoop.hive.ql.io.orc.OrcSerde | NULL | | InputFormat: | org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | NULL | | OutputFormat: | org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL | | Compressed: | No | NULL | | Num Buckets: | -1 | NULL | | Bucket Columns: | [] | NULL | | Sort Columns: | [] | NULL | | Storage Desc Params: | NULL | NULL | | | serialization.format | 1 | +-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--+ 26 rows selected (0.22 seconds) 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> set hive.server2.enable.doAs; +---------------------------------+--+ | set | +---------------------------------+--+ | hive.server2.enable.doAs=false | +---------------------------------+--+ Permalink Edit Delete tsaito Takahiko Saito added a comment - 3 days ago hdfs dir of temporary table is owned by hive as expected since hive.server2.enable.doAs=false: hdfs@hn0-hs21-h:~$ hdfs dfs -ls hdfs://hn0-hs21-h.hdinsight.net:8020/tmp/hive/hive/de2667ea-5bc1-4548-a4c4-97e46d6081f8/_tmp_space.db/8e8482c9-a2a4-4e8c-ad0f-2cd6e8fcdb4c Found 1 items -rwx------ 3 hive hdfs 47777 2016-02-02 23:02 hdfs://hn0-hs21-h.hdinsight.net:8020/tmp/hive/hive/de2667ea-5bc1-4548-a4c4-97e46d6081f8/_tmp_space.db/8e8482c9-a2a4-4e8c-ad0f-2cd6e8fcdb4c/000000_0 Not sure if it's related to the JIRA, but one thing I noticed was that dfs cmd throws error via beeline: 0: jdbc:hive2://zk2-hs21-h.hdinsight.net:2181> dfs -ls hdfs://hn0-hs21-h.hdinsight.net:8020/tmp/hive/hive/de2667ea-5bc1-4548-a4c4-97e46d6081f8/_tmp_space.db/8e8482c9-a2a4-4e8c-ad0f-2cd6e8fcdb4c; Error: Error while processing statement: Permission denied: user [hrt_qa] does not have privilege for [DFS] command (state=,code=1) Permalink Edit thejas Thejas Nair added a comment - 5 hours ago - edited Takahiko Saito The above DFS error would be from Ranger of SQL standard authorization. It is not related to the test failure. But you are right that it looks like a permission issue. The issue seen in HIVE-739 also exists in this part of Hive. Hive in MR mode launches a new child process to process the small table and create a hash table. This child process needs credentials from HDFS. However, in this setup, azure fs is the default file system. We should also get delegation token from all FS URIs listed under mapreduce.job.hdfs-servers config. There are hueristics around when map-join gets used. That is why you don't see it unless temp table is used (that uses ORC format and might have stats as well, while the original table is probaly text format). https://github.com/hortonworks/hive/blob/2.4-maint/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java needs change similar to webhcat change in HIVE-739 Permalink Edit sush Sushanth Sowmyan added a comment - 7 minutes ago Takahiko Saito, we should file an apache jira for this as well. Would you like to do so, so the bug report has attribution to you? Permalink Edit Delete tsaito Takahiko Saito added a comment - 4 minutes ago Sushanth Sowmyan I will file one and update with that JIRA. Comment People Assignee: sush Sushanth Sowmyan Assign to me Reporter: tsaito Takahiko Saito QEAssignee: Takahiko Saito Votes: 0 Watchers: 3 Stop watching this issue Dates Created: 3 days ago Updated: 4 minutes ago Who's Looking? Agile View on Board {noformat} hivesever2 log shows: {noformat} 2016-02-02 18:04:43,766 ERROR [HiveServer2-Background-Pool: Thread-517]: operation.Operation (SQLOperation.java:run(209)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {noformat}

    Apache's JIRA Issue Tracker | 1 year ago | Takahiko Saito
    org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Unable to export hive data into hbase table.

    Stack Overflow | 2 months ago | Yasodhara
    org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
  6. 0

    ES-Hadoop 5.0 Spark write issue - Hadoop and Elasticsearch - Discuss the Elastic Stack

    elastic.co | 3 weeks ago
    org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    6 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hive.service.cli.HiveSQLException

      Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

      at org.apache.hive.service.cli.operation.Operation.toSQLException()
    2. org.apache.hive
      SQLOperation$1$1.run
      1. org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
      2. org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
      3. org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
      4. org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
      4 frames
    3. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:415)
      2 frames
    4. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      1 frame
    5. org.apache.hive
      SQLOperation$1.run
      1. org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
      1 frame
    6. Java RT
      Thread.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      2. java.util.concurrent.FutureTask.run(FutureTask.java:262)
      3. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      4. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      5. java.lang.Thread.run(Thread.java:745)
      5 frames