org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread

hive-issues | Takuma Wakamori (JIRA) | 8 months ago
  1. 0

    "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000 HiveServer2 memory leak occurs. hive query {code} FROM hoge_tmp INSERT INTO TABLE hoge PARTITION (...) SELECT ... WHERE ... {code} stacktrace {code} org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577) at org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310) at org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559) at org.apache.tez.client.TezClient.start(TezClient.java:395) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1703) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1460) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1096) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154) ... 15 more {code}

    Apache's JIRA Issue Tracker | 9 months ago | Wataru Yukawa
    org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
  2. 0

    [jira] [Commented] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez

    hive-issues | 8 months ago | Takuma Wakamori (JIRA)
    org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
  3. 0

    netbout failed with OutOfMemoryError

    GitHub | 2 years ago | alevohin
    java.lang.OutOfMemoryError: unable to create new native thread
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    unable to create new native thread while running HttpServer

    Stack Overflow | 2 years ago | Erki M.
    java.lang.OutOfMemoryError: unable to create new native thread

  1. Reich 4 times, last 2 weeks ago
  2. Ajeet 46 times, last 4 months ago
  3. Diogo Jaym 1 times, last 6 months ago
  4. brunosatriani 2 times, last 6 months ago
  5. Bardh 4 times, last 7 months ago
1 more registered users
29 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.lang.OutOfMemoryError

    unable to create new native thread

    at java.lang.Thread.start0()
  2. Java RT
    Thread.start
    1. java.lang.Thread.start0(Native Method)
    2. java.lang.Thread.start(Thread.java:714)
    2 frames
  3. Apache Hadoop HDFS
    DistributedFileSystem$7.doCall
    1. org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238)
    2. org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753)
    3. org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
    4. org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
    5. org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
    6. org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
    6 frames
  4. Hadoop
    FileSystemLinkResolver.resolve
    1. org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    1 frame
  5. Apache Hadoop HDFS
    DistributedFileSystem.create
    1. org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
    2. org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
    2 frames
  6. Hadoop
    FileSystem.create
    1. org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    2. org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
    3. org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
    4. org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
    5. org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577)
    5 frames
  7. org.apache.tez
    TezClient.start
    1. org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310)
    2. org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559)
    3. org.apache.tez.client.TezClient.start(TezClient.java:395)
    3 frames
  8. Hive Query Language
    Driver.run
    1. org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
    2. org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
    3. org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151)
    4. org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
    5. org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
    6. org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1703)
    7. org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1460)
    8. org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
    9. org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
    10. org.apache.hadoop.hive.ql.Driver.run(Driver.java:1096)
    10 frames
  9. org.apache.hive
    TCLIService$Processor$ExecuteStatement.getResult
    1. org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
    2. org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
    3. org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
    4. org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410)
    5. org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391)
    6. org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
    7. org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
    8. org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
    9. org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
    9 frames
  10. Apache Thrift
    TBaseProcessor.process
    1. org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    2. org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    2 frames
  11. org.apache.hive
    TSetIpAddressProcessor.process
    1. org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
    1 frame
  12. Apache Thrift
    TThreadPoolServer$WorkerProcess.run
    1. org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
    1 frame
  13. Java RT
    Thread.run
    1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    3. java.lang.Thread.run(Thread.java:745)
    3 frames