java.lang.OutOfMemoryError: unable to create new native thread

Jenkins JIRA | porterhouse91 | 4 years ago
  1. 0

    [JENKINS-17630] java.lang.OutOfMemoryError: unable to create new native thread - Jenkins JIRA

    jenkins-ci.org | 3 months ago
    java.lang.OutOfMemoryError: unable to create new native thread
  2. 0

    We have about 200+ jobs that are polling Perforce from Jenkins and we noticed after a short period, Jenkins would start throwing OutOfMemoryErrors in the log whenever it tried to poll Perforce. After a period of a few hours, Jenkins stops becoming responsive to HTTP requests. Here's the error logged by Jenkins (project name replaced with $PROJECT_NAME): Apr 16, 2013 8:11:25 AM hudson.triggers.Trigger checkTriggers WARNING: hudson.triggers.SCMTrigger.run() failed for $PROJECT_NAME java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at hudson.util.SequentialExecutionQueue$QueueEntry.submit(SequentialExecutionQueue.java:108) at hudson.util.SequentialExecutionQueue$QueueEntry.access$100(SequentialExecutionQueue.java:95) at hudson.util.SequentialExecutionQueue.execute(SequentialExecutionQueue.java:66) at hudson.triggers.SCMTrigger.run(SCMTrigger.java:128) at hudson.triggers.SCMTrigger.run(SCMTrigger.java:102) at hudson.triggers.Trigger.checkTriggers(Trigger.java:261) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:209) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) When we investigated further, we found that Jenkins' Perforce plugin had created a large number of 'p4 client -i' processes but they were just sitting there. Some had become defunct over time. Here's a sample of the output I see from a 'ps -ef | grep jenkins' command. The actual output contains about 590 processes. jenkins 7257 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7262 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7273 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7278 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7281 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7282 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7283 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7285 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7287 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7288 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7289 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7290 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7291 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7292 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7294 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7295 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7298 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7302 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7306 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7309 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7312 1 3 06:08 ? 00:04:54 /usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --ajp13Port=8009 --debug=5 --handlerCountStartup=50 --handlerCountMax=100 --handlerCountMaxIdle=50 --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access_log jenkins 7313 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7325 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7331 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7356 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7357 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7358 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7359 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7490 7312 0 07:21 ? 00:00:00 /usr/local/sbin/p4 -s client -i jenkins 7495 7312 0 07:21 ? 00:00:00 /usr/local/sbin/p4 -s client -i jenkins 7499 7312 0 07:21 ? 00:00:00 /usr/local/sbin/p4 -s client -i As you can see, I suspected that this might be due to the same issue as JENKINS-16429, so I increased the handlers, but that only allowed additional threads to be created before Jenkins starts failing.

    Jenkins JIRA | 4 years ago | porterhouse91
    java.lang.OutOfMemoryError: unable to create new native thread
  3. 0

    We have about 200+ jobs that are polling Perforce from Jenkins and we noticed after a short period, Jenkins would start throwing OutOfMemoryErrors in the log whenever it tried to poll Perforce. After a period of a few hours, Jenkins stops becoming responsive to HTTP requests. Here's the error logged by Jenkins (project name replaced with $PROJECT_NAME): Apr 16, 2013 8:11:25 AM hudson.triggers.Trigger checkTriggers WARNING: hudson.triggers.SCMTrigger.run() failed for $PROJECT_NAME java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) at hudson.util.SequentialExecutionQueue$QueueEntry.submit(SequentialExecutionQueue.java:108) at hudson.util.SequentialExecutionQueue$QueueEntry.access$100(SequentialExecutionQueue.java:95) at hudson.util.SequentialExecutionQueue.execute(SequentialExecutionQueue.java:66) at hudson.triggers.SCMTrigger.run(SCMTrigger.java:128) at hudson.triggers.SCMTrigger.run(SCMTrigger.java:102) at hudson.triggers.Trigger.checkTriggers(Trigger.java:261) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:209) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) When we investigated further, we found that Jenkins' Perforce plugin had created a large number of 'p4 client -i' processes but they were just sitting there. Some had become defunct over time. Here's a sample of the output I see from a 'ps -ef | grep jenkins' command. The actual output contains about 590 processes. jenkins 7257 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7262 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7273 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7278 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7281 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7282 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7283 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7285 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7287 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7288 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7289 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7290 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7291 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7292 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7294 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7295 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7298 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7302 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7306 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7309 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7312 1 3 06:08 ? 00:04:54 /usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --ajp13Port=8009 --debug=5 --handlerCountStartup=50 --handlerCountMax=100 --handlerCountMaxIdle=50 --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access_log jenkins 7313 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7325 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7331 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7356 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7357 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7358 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7359 7312 0 07:21 ? 00:00:00 [p4] <defunct> jenkins 7490 7312 0 07:21 ? 00:00:00 /usr/local/sbin/p4 -s client -i jenkins 7495 7312 0 07:21 ? 00:00:00 /usr/local/sbin/p4 -s client -i jenkins 7499 7312 0 07:21 ? 00:00:00 /usr/local/sbin/p4 -s client -i As you can see, I suspected that this might be due to the same issue as JENKINS-16429, so I increased the handlers, but that only allowed additional threads to be created before Jenkins starts failing.

    Jenkins JIRA | 4 years ago | porterhouse91
    java.lang.OutOfMemoryError: unable to create new native thread
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Jenkins isn't accessible due to several errors

    Google Groups | 3 years ago | Mehdi Hayani Soujaa
    java.lang.OutOfMemoryError: unable to create new native thread
  6. 0

    OOME: unable to create new native thread from compile server

    YouTrack | 4 years ago
    java.lang.OutOfMemoryError: unable to create new native thread

    8 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      unable to create new native thread

      at java.lang.Thread.start0()
    2. Java RT
      AbstractExecutorService.submit
      1. java.lang.Thread.start0(Native Method)
      2. java.lang.Thread.start(Thread.java:691)
      3. java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
      4. java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
      5. java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
      5 frames
    3. Hudson
      SafeTimerTask.run
      1. hudson.util.SequentialExecutionQueue$QueueEntry.submit(SequentialExecutionQueue.java:108)
      2. hudson.util.SequentialExecutionQueue$QueueEntry.access$100(SequentialExecutionQueue.java:95)
      3. hudson.util.SequentialExecutionQueue.execute(SequentialExecutionQueue.java:66)
      4. hudson.triggers.SCMTrigger.run(SCMTrigger.java:128)
      5. hudson.triggers.SCMTrigger.run(SCMTrigger.java:102)
      6. hudson.triggers.Trigger.checkTriggers(Trigger.java:261)
      7. hudson.triggers.Trigger$Cron.doRun(Trigger.java:209)
      8. hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      8 frames
    4. Java RT
      TimerThread.run
      1. java.util.TimerThread.mainLoop(Timer.java:555)
      2. java.util.TimerThread.run(Timer.java:505)
      2 frames