hudson.model.Queue

Jenkins JIRA | Oleg Nenashev | 4 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1 Requests were getting blocked waiting on 0x00000004181e4520 {code} "Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000] java.lang.Thread.State: BLOCKED (on object monitor) at hudson.model.Queue.getItems(Queue.java:687) - waiting to lock <0x00000004181e4520> (a hudson.model.Queue) at hudson.model.Queue$CachedItemList.get(Queue.java:216) at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717) at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483) at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) {code} This seems to be waiting on Throttle Concurrent Builds code. Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace. {code} "Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000] java.lang.Thread.State: RUNNABLE at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875) at java.util.AbstractCollection.toArray(AbstractCollection.java:139) at java.util.ArrayList.<init>(ArrayList.java:164) at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141) - locked <0x000000041a79b778> (a java.util.HashMap) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90) at hudson.model.Queue.isBuildBlocked(Queue.java:937) at hudson.model.Queue.maintain(Queue.java:1006) - locked <0x00000004181e4520> (a hudson.model.Queue) at hudson.model.Queue$1.call(Queue.java:303) at hudson.model.Queue$1.call(Queue.java:300) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104) at java.lang.Thread.run(Thread.java:724) Locked ownable synchronizers: - None {code} CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins. (6021 = 0x1785) {code} PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6021 rcbuild_ 35 10 18.0g 6.2g 32m R 99.7 19.7 30:45.97 java {code} I have rolled back to Throttle Concurrent Builds 1.8 for now. Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.

    Jenkins JIRA | 3 years ago | Geoff Cummings
    hudson.model.Queue
  2. Speed up your debug routine!

    Automated exception search integrated into your IDE

  3. 0

    Manual removing of Docker build node with OnceRetentionStrategy causing deadlock with next stacktrace: {code} Found one Java-level deadlock: ============================= "Handling GET / from 192.168.10.18 : RequestHandlerThread[#255] View/index.jelly View/sidepanel.jelly": waiting to lock monitor 0x00007f4b9ae22bc8 (object 0x000000078713b8f0, a hudson.model.Queue), which is held by "jenkins.util.Timer [#3]" "jenkins.util.Timer [#3]": waiting to lock monitor 0x00007f4b9ae219e8 (object 0x0000000787158e00, a hudson.model.Hudson), which is held by "Executor #0 for bfe04df8701e@vm : executing project-release #125" "Executor #0 for bfe04df8701e@vm : executing project-release #125": waiting to lock monitor 0x00007f4b997e99b8 (object 0x0000000792523c40, a org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy), which is held by "jenkins.util.Timer [#3]" Java stack information for the threads listed above: =================================================== "Handling GET / from 192.168.10.18 : RequestHandlerThread[#255] View/index.jelly View/sidepanel.jelly": at hudson.model.Queue.getItems(Queue.java:693) - waiting to lock <0x000000078713b8f0> (a hudson.model.Queue) at hudson.model.Queue$CachedItemList.get(Queue.java:229) at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:723) at hudson.model.View.getApproximateQueueItemsQuickly(View.java:484) at sun.reflect.GeneratedMethodAccessor3580.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3500) at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:61) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:227) at jenkins.widgets.BuildQueueWidget.index.run(index.groovy:5) at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:74) at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:813) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "jenkins.util.Timer [#3]": at jenkins.model.Jenkins.removeNode(Jenkins.java:1703) - waiting to lock <0x0000000787158e00> (a hudson.model.Hudson) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:65) at hudson.slaves.CloudRetentionStrategy$1.run(CloudRetentionStrategy.java:67) at hudson.model.Queue._withLock(Queue.java:1035) - locked <0x000000078713b8f0> (a hudson.model.Queue) at hudson.model.Queue.withLock(Queue.java:983) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:58) - locked <0x0000000792523c40> (a org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:45) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:66) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "Executor #0 for bfe04df8701e@vm : executing project-release #125": at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:54) - waiting to lock <0x0000000792523c40> (a org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:45) at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:661) at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180) - locked <0x0000000787158f18> (a java.lang.Object) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1218) at jenkins.model.Jenkins.setNodes(Jenkins.java:1714) at jenkins.model.Jenkins.removeNode(Jenkins.java:1709) - locked <0x0000000787158e00> (a hudson.model.Hudson) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:65) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy.done(OnceRetentionStrategy.java:81) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy.taskCompleted(OnceRetentionStrategy.java:63) at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:296) at com.nirima.jenkins.plugins.docker.DockerComputer.taskCompleted(DockerComputer.java:34) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140) at hudson.model.Executor.run(Executor.java:253) Found 1 deadlock. {code}

    Jenkins JIRA | 2 years ago | Victor Antonovich
    hudson.model.Queue

    Root Cause Analysis

    1. hudson.model.Queue

      No message provided

      at hudson.model.Queue$CachedItemList.get()
    2. Hudson
      View.getApproximateQueueItemsQuickly
      1. hudson.model.Queue$CachedItemList.get(Queue.java:196)
      2. hudson.model.Queue.getApproximateItemsQuickly(Queue.java:671)
      3. hudson.model.View.getApproximateQueueItemsQuickly(View.java:457)
      3 frames
    3. Java RT
      GeneratedMethodAccessor248.invoke
      1. sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
      1 frame