java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds

Jenkins JIRA | Luis Silva | 9 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    With the latest plugin version (1.4) my tasks fail to run on the ecs slaves. They sit waiting for executors forever. I made sure I had two servers with plenty of resources connected to ECS. It looks like the task is assigned to a node but then immediately disconnects. After downgrading to version 1.3 it all started working perfectly again. I tested it with the same number of servers, same tasks, same jenkins versions, etc. See the logs below for both versions. Version 1.4 failures: {quote} Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 3,767mb of free memory. 3,000mb are required Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 1,024 units of free cpu. 512 units are required Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e94238750e85 Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure to run task with definition arn:aws:ecs:eu-west-1:AWS_ACC:task-definition/jenkins-slave:78 on ECS cluster arn:aws:ecs:eu-west-1:AWS_ACC:cluster/JenkinsSlaves Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure reason=RESOURCE:MEMORY, arn=arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent ECS Slave BackendTests failed to launch java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent ECS Slave BackendTests failed to launch hudson.AbortException: Failed to run slave container JenkinsSlavesECS-16e94238750e85 at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:157) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:250) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.816 Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: -0.184 Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources .......... Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Resources found in instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601: [{Name: CPU,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 1024,StringSetValue: []}, {Name: MEMORY,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 3767,StringSetValue: []}, {Name: PORTS,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: [22, 2376, 2375, 51678]}, {Name: PORTS_UDP,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: []}] Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 3,767mb of free memory. 3,000mb are required Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 1,024 units of free cpu. 512 units are required Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e9448c0da1df Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Slave JenkinsSlavesECS-16e9448c0da1df - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call WARNING: ECS Slave JenkinsSlavesECS-16e9448c0da1df (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da) not connected since 7/6/16 8:37 AM seconds Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained INFO: Started termination Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained INFO: Completed termination Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins _cleanUpDisconnectComputers INFO: Starting node disconnection {quote} After downgrade to version 1.3, working well: {quote} Jul 06, 2016 8:40:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.303 Jul 06, 2016 8:40:06 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e965bf8c0ffe Jul 06, 2016 8:40:07 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Slave JenkinsSlavesECS-16e965bf8c0ffe - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c Jul 06, 2016 8:40:11 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #1 from /XX.XX.XX.XX:YY Jul 06, 2016 8:40:15 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave JenkinsSlavesECS-16e965bf8c0ffe (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c) connected Jul 06, 2016 8:40:16 AM hudson.slaves.NodeProvisioner$2 run INFO: ECS Slave BackendTests provisioning successfully completed. We have now 3 computer(s) {quote}

    Jenkins JIRA | 9 months ago | Luis Silva
    java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds
  2. 0

    [JIRA] (JENKINS-36468) Tasks fail to run on ECS after upgrade to 1.4

    Google Groups | 9 months ago | lu...@vvoosh.com (JIRA)
    java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-yyyyyy (ecs task arn:aws:ecs:eu-west-1:ACCOUNT_ID:task/ed743e1e-7be1-455b-a01d-zzzzzz) not connected since 8/10/16 4:31 PM seconds
  3. 0

    [JIRA] (JENKINS-41199) Provisioned agent ECS Slave ecs-quitest failed to launch

    Google Groups | 2 months ago | tiend...@gmail.com (JIRA)
    java.lang.IllegalStateException: ECS Slave ecs-cloud-364b79cfdce9 (ecs task arn:aws:ecs:eu-central-1:280052068306:task/a65941bb-c874-43a6-a008-bcefe37ba60b) not connected since 1/19/17 3:27 AM seconds
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    How can I cope with "IllegalStateException: BeanFactory not initialized or already closed"?

    Stack Overflow | 6 years ago | user569825
    java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
  6. 0

    issues in spring petclinic applicaiton

    Stack Overflow | 4 years ago | user2216702
    java.lang.IllegalStateException: ApplicationEventMulticaster not initialized - c all 'refresh' before multicasting events via the context: Root WebApplicationCon text: startup date [Wed Mar 27 11:31:56 CDT 2013]; root of context hierarchy at org.springframework.context.support.AbstractApplic ationContext.getApp licationEventMulticaster(AbstractApplicationContex t.java:347) [spring-context-3. 2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.context.support.AbstractApplic ationContext.publis hEvent(AbstractApplicationContext.java:334) [spring-context-3.2.2.RELEASE.jar:3. 2.2.RELEASE] at org.springframework.context.support.AbstractApplic ationContext.doClos e(AbstractApplicationContext.java:1049) [spring-context-3.2.2.RELEASE.jar:3.2.2. RELEASE] at org.springframework.context.support.AbstractApplic ationContext.close( AbstractApplicationContext.java:1010) [spring-context-3.2.2.RELEASE.jar:3.2.2.RE LEASE] at org.springframework.web.context.ContextLoader.clos eWebApplicationCont ext(ContextLoader.java:558) [spring-web-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.web.context.ContextLoaderListe ner.contextDestroye d(ContextLoaderListener.java:143) [spring-web-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.apache.catalina.core.StandardContext.listenerS top(StandardContext .java:4831) [tomcat-embed-core-7.0.30.jar:7.0.30] at org.apache.catalina.core.StandardContext.stopInter nal(StandardContext .java:5478) [tomcat-embed-core-7.0.30.jar:7.0.30] at org.apache.catalina.util.LifecycleBase.stop(Lifecy cleBase.java:232) [ tomcat-embed-core-7.0.30.jar:7.0.30] at org.apache.catalina.util.LifecycleBase.start(Lifec ycleBase.java:160) [tomcat-embed-core-7.0.30.jar:7.0.30] at org.apache.catalina.core.ContainerBase$StartChild. call(ContainerBase. java:1559) [tomcat-embed-core-7.0.30.jar:7.0.30] at org.apache.catalina.core.ContainerBase$StartChild. call(ContainerBase. java:1549) [tomcat-embed-core-7.0.30.jar:7.0.30] at java.util.concurrent.FutureTask$Sync.innerRun(Futu reTask.java:303) [n a:1.6.0_20] at java.util.concurrent.FutureTask.run(FutureTask.jav a:138) [na:1.6.0_20 ] at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExec utor.java:886) [na:1.6.0_20] at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor .java:908) [na:1.6.0_20]

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalStateException

      ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds

      at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call()
    2. com.cloudbees.jenkins
      ECSCloud$ProvisioningCallback.call
      1. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278)
      2. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
      2 frames
    3. jenkins.util
      ContextResettingExecutorService$2.call
      1. jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      1 frame
    4. Java RT
      Thread.run
      1. java.util.concurrent.FutureTask.run(FutureTask.java:262)
      2. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      3. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      4. java.lang.Thread.run(Thread.java:745)
      4 frames