java.lang.RuntimeException: unable to create new native thread

Atlassian JIRA | Thiago Bomfim [Atlassian] | 1 year ago
  1. 0

    [BSERV-7616] executor.max.threads is too big for instances with too many CPUs - Atlassian JIRA

    atlassian.com | 1 year ago
    java.lang.RuntimeException: unable to create new native thread
  2. 0

    h3. Problem statement After fixing STASH-6868 which describes: {quote} The default value of {{executor.max.threads}} is 100 which is fine for average sized instances, but probably too small for very large instances. This limit is often the first one to be hit on instances with very high (especially SSH clone) hosting workload. The value should be increased, preferably scaling with instance size. {quote} We introduced a formula for {{executor.max.threads=20*cpu+100}} as can be seen on [Stash config properties - Executor|https://confluence.atlassian.com/display/STASH/Stash+config+properties#Stashconfigproperties-Executor]. The problem is that this value could be too big for big instances, generating errors like: {noformat} Exception in thread "ssh:thread-5" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat} Or {noformat} 2015-03-24 18:46:39,850 ERROR [AtlassianEvent::thread-23] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.crowd.event.user.UserAttributeStoredEvent@d1e8fd3f] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.stash.internal.license.DefaultLicensedUserCountCache.onUserUpdatedEvent(com.atlassian.crowd.event.user.UserUpdatedEvent), listener=com.atlassian.stash.internal.license.DefaultLicensedUserCountCache@1301e2fa}] java.lang.RuntimeException: unable to create new native thread at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) [atlassian-event-2.3.5.jar:na] at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) [atlassian-event-2.3.5.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) [guava-11.0.2-atlassian-02.jar:na] at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107) [atlassian-event-2.3.5.jar:na] {noformat} We should check the number of CPUs available and have a ceiling for that value to avoid such scenarios. h3. Workaround Set in your {{stash-config.properties}} a fixed value for {{executor.max.threads}} which is lower and is reasonable to your system. This will overwrite the formula we ship by default once you restart Stash.

    Atlassian JIRA | 1 year ago | Thiago Bomfim [Atlassian]
    java.lang.RuntimeException: unable to create new native thread
  3. 0

    h3. Problem statement After fixing STASH-6868 which describes: {quote} The default value of {{executor.max.threads}} is 100 which is fine for average sized instances, but probably too small for very large instances. This limit is often the first one to be hit on instances with very high (especially SSH clone) hosting workload. The value should be increased, preferably scaling with instance size. {quote} We introduced a formula for {{executor.max.threads=20*cpu+100}} as can be seen on [Stash config properties - Executor|https://confluence.atlassian.com/display/STASH/Stash+config+properties#Stashconfigproperties-Executor]. The problem is that this value could be too big for big instances, generating errors like: {noformat} Exception in thread "ssh:thread-5" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat} Or {noformat} 2015-03-24 18:46:39,850 ERROR [AtlassianEvent::thread-23] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.crowd.event.user.UserAttributeStoredEvent@d1e8fd3f] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.stash.internal.license.DefaultLicensedUserCountCache.onUserUpdatedEvent(com.atlassian.crowd.event.user.UserUpdatedEvent), listener=com.atlassian.stash.internal.license.DefaultLicensedUserCountCache@1301e2fa}] java.lang.RuntimeException: unable to create new native thread at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) [atlassian-event-2.3.5.jar:na] at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) [atlassian-event-2.3.5.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) [guava-11.0.2-atlassian-02.jar:na] at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107) [atlassian-event-2.3.5.jar:na] {noformat} We should check the number of CPUs available and have a ceiling for that value to avoid such scenarios. h3. Workaround Set in your {{stash-config.properties}} a fixed value for {{executor.max.threads}} which is lower and is reasonable to your system. This will overwrite the formula we ship by default once you restart Stash.

    Atlassian JIRA | 1 year ago | Thiago Bomfim [Atlassian]
    java.lang.RuntimeException: unable to create new native thread
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    GitHub comment 83#143812446

    GitHub | 1 year ago | wadahiro
    java.lang.RuntimeException: You are not permitted to access this resource
  6. 0

    [WEBHOOKS-95] OsgiWebHookProviderWebHookRegistry has flawed locking logic - Ecosystem JIRA

    atlassian.net | 12 months ago
    java.lang.RuntimeException: attempt to unlock read lock, not locked by current thread

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.RuntimeException

      unable to create new native thread

      at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke()
    2. com.atlassian.event
      AsynchronousAbleEventDispatcher$1$1.run
      1. com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54)[atlassian-event-2.3.5.jar:na]
      2. com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)[atlassian-event-2.3.5.jar:na]
      2 frames
    3. Guava
      MoreExecutors$SameThreadExecutorService.execute
      1. com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)[guava-11.0.2-atlassian-02.jar:na]
      1 frame
    4. com.atlassian.event
      AsynchronousAbleEventDispatcher.dispatch
      1. com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)[atlassian-event-2.3.5.jar:na]
      1 frame