javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException

Atlassian JIRA | David Corley | 5 years ago
  1. 0

    Today we experienced an issue where our Bamboo server ran out of disk space. It came to our attention due to to the fact that all of the remote agents went offline at the same time. We had set up the agent wrapper.conf to continue to try and reconnect to the server for 6 hours after an initial connection failure. When the server ran out of disk space we saw the following in the agent logs: {noformat} INFO | jvm 1 | 2011/08/25 03:25:05 | org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 1 | 2011/08/25 03:25:05 | Caused by: javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1329) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1314) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1758) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:536) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:513) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate$2.doInJms(JmsTemplate.java:479) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:430) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:588) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:576) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.v2.build.agent.remote.sender.JmsTemplateMessageSender.send(JmsTemplateMessageSender.java:33) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.v2.build.agent.remote.heartbeat.AgentHeartBeatJob.execute(AgentHeartBeatJob.java:51) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.quartz.core.JobRunShell.run(JobRunShell.java:202) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:38) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) INFO | jvm 1 | 2011/08/25 03:25:05 | Caused by: java.lang.Throwable: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device {noformat} Even after we freed 50GB on the server disk, we saw the following in the agent logs: {noformat} INFO | jvm 86 | 2011/08/25 10:31:31 | 2011-08-25 10:31:31,663 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' was unsuccessful. Run 1 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 86 | 2011/08/25 10:31:31 | 2011-08-25 10:31:31,664 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Waiting 1000ms before retrying... INFO | jvm 86 | 2011/08/25 10:31:32 | 2011-08-25 10:31:32,674 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' was unsuccessful. Run 2 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 86 | 2011/08/25 10:31:32 | 2011-08-25 10:31:32,674 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Waiting 2000ms before retrying... INFO | jvm 86 | 2011/08/25 10:31:34 | 2011-08-25 10:31:34,683 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' finished successfully during retry number 2 INFO | jvm 86 | 2011/08/25 11:07:16 | 2011-08-25 11:07:16,646 INFO [QuartzScheduler_Worker-10] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:10:46 | 2011-08-25 11:10:46,650 INFO [QuartzScheduler_Worker-7] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:10:51 | 2011-08-25 11:10:51,647 INFO [QuartzScheduler_Worker-1] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:12:01 | 2011-08-25 11:12:01,651 INFO [QuartzScheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... {noformat} Attempting to restart one of the agents we saw: {noformat} INFO | jvm 2 | 2011/08/25 04:29:09 | 2011-08-25 04:29:09,370 WARN [Thread-1] [JmsClientInterceptor] Remote access error: ReflectiveMethodInvocation: public abstract com.atlassian.bamboo.buildqueue.PipelineDefinition com.atlassian.bamboo.buildqueue.manager.RemoteAgentManager.registerAgent(com.atlassian.bamboo.buildqueue.PipelineDefinition) throws com.atlassian.bamboo.license.BambooLicenseException; target is null INFO | jvm 2 | 2011/08/25 04:29:09 | javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 2 | 2011/08/25 04:29:09 | Exiting due to fatal exception. INFO | jvm 2 | 2011/08/25 04:29:09 | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean' defined in class path resource [applicationContextRemoteAgentStartup.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 2 | 2011/08/25 04:29:09 | 2011-08-25 04:29:09,371 FATAL [Thread-1] [AgentBootstrap] Exiting due to fatal exception. INFO | jvm 2 | 2011/08/25 04:29:09 | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean' defined in class path resource [applicationContextRemoteAgentStartup.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) ERROR | wrapper | 2011/08/25 04:29:09 | JVM exited unexpectedly. {noformat} It was only when we restarted the server that the agents reconnected successfully (we didn't restart any of the agents). It seems that when the disk space error occurred and the agents all took themselves offline, Bamboo made it impossible for them to come back online even though they were trying continuously to do so. This seems like a bug relating to Bamboo's management of connection state for the remote agents.

    Atlassian JIRA | 5 years ago | David Corley
    javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException
  2. 0

    Today we experienced an issue where our Bamboo server ran out of disk space. It came to our attention due to to the fact that all of the remote agents went offline at the same time. We had set up the agent wrapper.conf to continue to try and reconnect to the server for 6 hours after an initial connection failure. When the server ran out of disk space we saw the following in the agent logs: {noformat} INFO | jvm 1 | 2011/08/25 03:25:05 | org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 1 | 2011/08/25 03:25:05 | Caused by: javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1329) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1314) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1758) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:536) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:513) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate$2.doInJms(JmsTemplate.java:479) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:430) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:588) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:576) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.v2.build.agent.remote.sender.JmsTemplateMessageSender.send(JmsTemplateMessageSender.java:33) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.v2.build.agent.remote.heartbeat.AgentHeartBeatJob.execute(AgentHeartBeatJob.java:51) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.quartz.core.JobRunShell.run(JobRunShell.java:202) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:38) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) INFO | jvm 1 | 2011/08/25 03:25:05 | Caused by: java.lang.Throwable: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device {noformat} Even after we freed 50GB on the server disk, we saw the following in the agent logs: {noformat} INFO | jvm 86 | 2011/08/25 10:31:31 | 2011-08-25 10:31:31,663 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' was unsuccessful. Run 1 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 86 | 2011/08/25 10:31:31 | 2011-08-25 10:31:31,664 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Waiting 1000ms before retrying... INFO | jvm 86 | 2011/08/25 10:31:32 | 2011-08-25 10:31:32,674 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' was unsuccessful. Run 2 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 86 | 2011/08/25 10:31:32 | 2011-08-25 10:31:32,674 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Waiting 2000ms before retrying... INFO | jvm 86 | 2011/08/25 10:31:34 | 2011-08-25 10:31:34,683 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' finished successfully during retry number 2 INFO | jvm 86 | 2011/08/25 11:07:16 | 2011-08-25 11:07:16,646 INFO [QuartzScheduler_Worker-10] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:10:46 | 2011-08-25 11:10:46,650 INFO [QuartzScheduler_Worker-7] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:10:51 | 2011-08-25 11:10:51,647 INFO [QuartzScheduler_Worker-1] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:12:01 | 2011-08-25 11:12:01,651 INFO [QuartzScheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... {noformat} Attempting to restart one of the agents we saw: {noformat} INFO | jvm 2 | 2011/08/25 04:29:09 | 2011-08-25 04:29:09,370 WARN [Thread-1] [JmsClientInterceptor] Remote access error: ReflectiveMethodInvocation: public abstract com.atlassian.bamboo.buildqueue.PipelineDefinition com.atlassian.bamboo.buildqueue.manager.RemoteAgentManager.registerAgent(com.atlassian.bamboo.buildqueue.PipelineDefinition) throws com.atlassian.bamboo.license.BambooLicenseException; target is null INFO | jvm 2 | 2011/08/25 04:29:09 | javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 2 | 2011/08/25 04:29:09 | Exiting due to fatal exception. INFO | jvm 2 | 2011/08/25 04:29:09 | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean' defined in class path resource [applicationContextRemoteAgentStartup.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 2 | 2011/08/25 04:29:09 | 2011-08-25 04:29:09,371 FATAL [Thread-1] [AgentBootstrap] Exiting due to fatal exception. INFO | jvm 2 | 2011/08/25 04:29:09 | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean' defined in class path resource [applicationContextRemoteAgentStartup.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) ERROR | wrapper | 2011/08/25 04:29:09 | JVM exited unexpectedly. {noformat} It was only when we restarted the server that the agents reconnected successfully (we didn't restart any of the agents). It seems that when the disk space error occurred and the agents all took themselves offline, Bamboo made it impossible for them to come back online even though they were trying continuously to do so. This seems like a bug relating to Bamboo's management of connection state for the remote agents.

    Atlassian JIRA | 5 years ago | David Corley
    javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException
  3. 0

    My company recently purchased most the Atlassian suite, including Bamboo with a few remote agents. I started a remote agent last week and it was working. A few days later and I cannot get it to start up due to exception: INFO | jvm 41 | 2011/04/04 11:40:15 | 2011-04-04 11:40:15,208 WARN [Thread-1] [JmsClientInterceptor] Remote access error: ReflectiveMethodInvocation: public abstract com.atlassian.bamboo.buildqueue.PipelineDefinition com.atlassian.bamboo.buildqueue.manager.RemoteAgentManager.registerAgent(com.atlassian.bamboo.buildqueue.PipelineDefinition) throws com.atlassian.bamboo.license.BambooLicenseException; target is null INFO | jvm 41 | 2011/04/04 11:40:15 | javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 41 | 2011/04/04 11:40:15 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 41 | 2011/04/04 11:40:15 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 41 | 2011/04/04 11:40:15 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 41 | 2011/04/04 11:40:15 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 41 | 2011/04/04 11:40:15 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 41 | 2011/04/04 11:40:15 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 41 | 2011/04/04 11:40:15 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 41 | 2011/04/04 11:40:15 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) Please let me know what further info would help investigation.

    Atlassian JIRA | 6 years ago | Kallin nagelberg
    javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    My company recently purchased most the Atlassian suite, including Bamboo with a few remote agents. I started a remote agent last week and it was working. A few days later and I cannot get it to start up due to exception: INFO | jvm 41 | 2011/04/04 11:40:15 | 2011-04-04 11:40:15,208 WARN [Thread-1] [JmsClientInterceptor] Remote access error: ReflectiveMethodInvocation: public abstract com.atlassian.bamboo.buildqueue.PipelineDefinition com.atlassian.bamboo.buildqueue.manager.RemoteAgentManager.registerAgent(com.atlassian.bamboo.buildqueue.PipelineDefinition) throws com.atlassian.bamboo.license.BambooLicenseException; target is null INFO | jvm 41 | 2011/04/04 11:40:15 | javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 41 | 2011/04/04 11:40:15 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 41 | 2011/04/04 11:40:15 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 41 | 2011/04/04 11:40:15 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 41 | 2011/04/04 11:40:15 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 41 | 2011/04/04 11:40:15 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 41 | 2011/04/04 11:40:15 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 41 | 2011/04/04 11:40:15 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 41 | 2011/04/04 11:40:15 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 41 | 2011/04/04 11:40:15 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) Please let me know what further info would help investigation.

    Atlassian JIRA | 6 years ago | Kallin nagelberg
    javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException
  6. 0

    Seen the following exception on the agent and I believe they indicate a serious issue (an unhandled exception) in the agent as it 'went offline': {noformat} 2014-04-03 03:42:22,089 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [RemoteExecutionPhaseServiceImpl] A2. JIRA Master Tier 2 - Acceptance Tests and Suite Level Acceptance Tests - Deploy to Unicorn DEV instance # 492 (MASTERTWO-ATSLAT-JOB1-492) execution started 2014-04-03 03:42:22,119 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [TaskExecutorImpl] Skipping execution of task 'Task for destroying instance. Use only when want a clean run. Otherwise keep this task disabled. ' as it is disabled 2014-04-03 03:42:22,168 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [TaskExecutorImpl] Starting task 'Initial setup' of type 'com.atlassian.bamboo.plugins.bamboonicorn:setupunicorn' 2014-04-03 03:58:15,717 WARN [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [JmsClientInterceptor] Remote access error: ReflectiveMethodInvocation: public abstract com.atlassian.bamboo.configuration.AdministrationConfig uration com.atlassian.bamboo.configuration.AdministrationConfigurationAccessor.getAdministrationConfiguration(); target is null javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy2.getAdministrationConfiguration(Unknown Source) at com.atlassian.bamboo.utils.BambooUrl.withBaseUrlFromConfiguration(BambooUrl.java:55) at com.atlassian.bamboo.variable.CustomVariableContextImpl.addVariables(CustomVariableContextImpl.java:505) at com.atlassian.bamboo.variable.CustomVariableContextImpl.getCommonBuildVariables(CustomVariableContextImpl.java:464) at com.atlassian.bamboo.variable.CustomVariableContextImpl.filterVariableContextByName(CustomVariableContextImpl.java:334) at com.atlassian.bamboo.variable.CustomVariableContextImpl.substituteBambooVariables(CustomVariableContextImpl.java:197) at com.atlassian.bamboo.variable.CustomVariableContextImpl.substituteString(CustomVariableContextImpl.java:153) at com.atlassian.bamboo.task.TaskExecutorImpl.substituteConfigurationVariables(TaskExecutorImpl.java:281) at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:182) at com.atlassian.bamboo.task.TaskExecutorImpl.execute(TaskExecutorImpl.java:98) at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:75) at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:181) at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:101) at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:108) at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) at java.lang.Thread.run(Thread.java:662) 2014-04-03 03:58:15,719 FATAL [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [ExecuteBuildTask] Failed to execute the build 'MASTERTWO-ATSLAT-JOB1-492' org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy2.getAdministrationConfiguration(Unknown Source) at com.atlassian.bamboo.utils.BambooUrl.withBaseUrlFromConfiguration(BambooUrl.java:55) at com.atlassian.bamboo.variable.CustomVariableContextImpl.addVariables(CustomVariableContextImpl.java:505) at com.atlassian.bamboo.variable.CustomVariableContextImpl.getCommonBuildVariables(CustomVariableContextImpl.java:464) at com.atlassian.bamboo.variable.CustomVariableContextImpl.filterVariableContextByName(CustomVariableContextImpl.java:334) at com.atlassian.bamboo.variable.CustomVariableContextImpl.substituteBambooVariables(CustomVariableContextImpl.java:197) at com.atlassian.bamboo.variable.CustomVariableContextImpl.substituteString(CustomVariableContextImpl.java:153) at com.atlassian.bamboo.task.TaskExecutorImpl.substituteConfigurationVariables(TaskExecutorImpl.java:281) at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:182) at com.atlassian.bamboo.task.TaskExecutorImpl.execute(TaskExecutorImpl.java:98) at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:75) at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:181) at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:101) at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:108) at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) at java.lang.Thread.run(Thread.java:662) 2014-04-03 03:58:15,796 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [RemoteExecutionPhaseServiceImpl] A2. JIRA Master Tier 2 - Acceptance Tests and Suite Level Acceptance Tests - Deploy to Unicorn DEV instance #492 (MASTERTWO-ATSLAT-JOB1-492) execution finished 2014-04-03 03:58:15,818 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [RemoteResultProcessor] Build A2. JIRA Master Tier 2 - Acceptance Tests and Suite Level Acceptance Tests - Deploy to Unicorn DEV instance #492 (MASTERTWO-ATSLAT-JOB1-492) completed on Bamboo agent. Sending results to server. 2014-04-03 03:58:15,867 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [DefaultBuildAgent] Setting context to null on agent-02-05.buildeng.atlassian.com/869204611 2014-04-03 03:58:15,867 INFO [3-BAM::agent-02-05.buildeng.atlassian.com::Agent:pool-6-thread-1] [BuildAgentControllerImpl] Agent 'agent-02-05.buildeng.atlassian.com' ready to take build from queue... {noformat} The build log contained the following output: {noformat} ... busy doing work ... simple 03-Apr-2014 03:53:15 Finished task 'Initial setup' error 03-Apr-2014 03:58:15 Failed to execute the build 'MASTERTWO-ATSLAT-JOB1-492': Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException simple 03-Apr-2014 03:58:15 Finalising the build... simple 03-Apr-2014 03:58:15 Stopping timer. simple 03-Apr-2014 03:58:15 Build MASTERTWO-ATSLAT-JOB1-492 completed. {noformat} The _Initial setup_ task was not the last task for that job, only number 2 of 4. It would be nice if the Bamboo agent would do a retry and fail after 3 or 5 failed tries. Also, it would be nice it there would a meaningful message in the build log (on error level) that the communication with the server failed.

    Atlassian JIRA | 3 years ago | Martin Meinhold
    javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. javax.jms.JMSException

      edu.emory.mathcs.backport.java.util.concurrent.TimeoutException

      at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException()
    2. Lingo
      MultiplexingRequestor.request
      1. org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205)
      2. org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133)
      2 frames
    3. com.atlassian.bamboo
      BambooMultiplexingRequestor.request
      1. com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66)
      1 frame
    4. Lingo
      JmsClientInterceptor.invoke
      1. org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138)
      1 frame
    5. Spring AOP
      JdkDynamicAopProxy.invoke
      1. org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      2. org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      2 frames
    6. Unknown
      $Proxy10.registerAgent
      1. $Proxy10.registerAgent(Unknown Source)
      1 frame
    7. com.atlassian.bamboo
      AgentRegistrationBean.registerAgent
      1. com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41)
      1 frame
    8. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    9. Spring Beans
      DefaultListableBeanFactory.preInstantiateSingletons
      1. org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237)
      2. org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203)
      3. org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167)
      4. org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427)
      5. org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249)
      6. org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155)
      7. org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246)
      8. org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160)
      9. org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291)
      9 frames
    10. Spring Context
      ClassPathXmlApplicationContext.<init>
      1. org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352)
      2. org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122)
      3. org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88)
      3 frames
    11. com.atlassian.bamboo
      AgentRunner.run
      1. com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170)
      2. com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52)
      3. com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21)
      3 frames
    12. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:662)
      1 frame