java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-yyyyyy (ecs task arn:aws:ecs:eu-west-1:ACCOUNT_ID:task/ed743e1e-7be1-455b-a01d-zzzzzz) not connected since 8/10/16 4:31 PM seconds

Google Groups | lu...@vvoosh.com (JIRA) | 5 months ago
  1. 0

    [JIRA] (JENKINS-36468) Tasks fail to run on ECS after upgrade to 1.4

    Google Groups | 5 months ago | lu...@vvoosh.com (JIRA)
    java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-yyyyyy (ecs task arn:aws:ecs:eu-west-1:ACCOUNT_ID:task/ed743e1e-7be1-455b-a01d-zzzzzz) not connected since 8/10/16 4:31 PM seconds
  2. 0

    With the latest plugin version (1.4) my tasks fail to run on the ecs slaves. They sit waiting for executors forever. I made sure I had two servers with plenty of resources connected to ECS. It looks like the task is assigned to a node but then immediately disconnects. After downgrading to version 1.3 it all started working perfectly again. I tested it with the same number of servers, same tasks, same jenkins versions, etc. See the logs below for both versions. Version 1.4 failures: {quote} Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 3,767mb of free memory. 3,000mb are required Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 1,024 units of free cpu. 512 units are required Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e94238750e85 Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure to run task with definition arn:aws:ecs:eu-west-1:AWS_ACC:task-definition/jenkins-slave:78 on ECS cluster arn:aws:ecs:eu-west-1:AWS_ACC:cluster/JenkinsSlaves Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure reason=RESOURCE:MEMORY, arn=arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent ECS Slave BackendTests failed to launch java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent ECS Slave BackendTests failed to launch hudson.AbortException: Failed to run slave container JenkinsSlavesECS-16e94238750e85 at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:157) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:250) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.816 Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: -0.184 Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources .......... Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Resources found in instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601: [{Name: CPU,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 1024,StringSetValue: []}, {Name: MEMORY,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 3767,StringSetValue: []}, {Name: PORTS,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: [22, 2376, 2375, 51678]}, {Name: PORTS_UDP,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: []}] Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 3,767mb of free memory. 3,000mb are required Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 1,024 units of free cpu. 512 units are required Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e9448c0da1df Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Slave JenkinsSlavesECS-16e9448c0da1df - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call WARNING: ECS Slave JenkinsSlavesECS-16e9448c0da1df (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da) not connected since 7/6/16 8:37 AM seconds Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained INFO: Started termination Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained INFO: Completed termination Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins _cleanUpDisconnectComputers INFO: Starting node disconnection {quote} After downgrade to version 1.3, working well: {quote} Jul 06, 2016 8:40:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.303 Jul 06, 2016 8:40:06 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e965bf8c0ffe Jul 06, 2016 8:40:07 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Slave JenkinsSlavesECS-16e965bf8c0ffe - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c Jul 06, 2016 8:40:11 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #1 from /XX.XX.XX.XX:YY Jul 06, 2016 8:40:15 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave JenkinsSlavesECS-16e965bf8c0ffe (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c) connected Jul 06, 2016 8:40:16 AM hudson.slaves.NodeProvisioner$2 run INFO: ECS Slave BackendTests provisioning successfully completed. We have now 3 computer(s) {quote}

    Jenkins JIRA | 5 months ago | Luis Silva
    java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds
  3. 0

    Hi, my Jenkins server is showing "Executor Failure". Restarting the server doesn't help. {code:java} java.lang.IllegalStateException: cannot create a build with number 195 since that (or higher) is already in use among [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727, 1728] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:192) at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178) at hudson.model.AbstractProject.newBuild(AbstractProject.java:1011) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1210) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor$1.call(Executor.java:328) at hudson.model.Executor$1.call(Executor.java:310) at hudson.model.Queue._withLock(Queue.java:1251) at hudson.model.Queue.withLock(Queue.java:1189) at hudson.model.Executor.run(Executor.java:310) {code}

    Jenkins JIRA | 1 year ago | Chin Boon Oh
    java.lang.IllegalStateException: cannot create a build with number 195 since that (or higher) is already in use among [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727, 1728]
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hi, my Jenkins server is showing "Executor Failure". Restarting the server doesn't help. {code:java} java.lang.IllegalStateException: cannot create a build with number 195 since that (or higher) is already in use among [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727, 1728] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:192) at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178) at hudson.model.AbstractProject.newBuild(AbstractProject.java:1011) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1210) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor$1.call(Executor.java:328) at hudson.model.Executor$1.call(Executor.java:310) at hudson.model.Queue._withLock(Queue.java:1251) at hudson.model.Queue.withLock(Queue.java:1189) at hudson.model.Executor.run(Executor.java:310) {code}

    Jenkins JIRA | 1 year ago | Chin Boon Oh
    java.lang.IllegalStateException: cannot create a build with number 195 since that (or higher) is already in use among [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727, 1728]
  6. 0

    [JIRA] [core] (JENKINS-28915) Thread Death and build number is already in use

    Google Groups | 1 year ago | michael....@hella.com (JIRA)
    java.lang.IllegalStateException: cannot create a build with number 18 since that (or higher) is already in use among <span class="error">[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 195]</span>

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalStateException

      ECS Slave JenkinsSlavesECS-yyyyyy (ecs task arn:aws:ecs:eu-west-1:ACCOUNT_ID:task/ed743e1e-7be1-455b-a01d-zzzzzz) not connected since 8/10/16 4:31 PM seconds

      at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call()
    2. com.cloudbees.jenkins
      ECSCloud$ProvisioningCallback.call
      1. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278)
      2. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
      2 frames
    3. jenkins.util
      ContextResettingExecutorService$2.call
      1. jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      1 frame
    4. Java RT
      ThreadPoolExecutor$Worker.run
      1. java.util.concurrent.FutureTask.run(FutureTask.java:262)
      2. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      3. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      3 frames