com.amazonaws.AmazonServiceException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • We use ec2-plugin to start on-demand slaves and terminate them after 30m of idle time. This was working fine, until last week when we noticed a full build queue and an empty list of build executors. On EC2, there actually was a slave running (i-1e42a2a7). Below is the relevant fragment from the log: {code} WARNING: Failed to fetch EC2 instance: i-1e42a2a7 com.amazonaws.AmazonServiceException: The service is unavailable. Please try again shortly. (Service: AmazonEC2; Status Code: 503; Error Code: Unavailable; Re-rw-r---quest ID: ff9e539b-f87f-4143-80b6-e11fbf61378e) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11768) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5899) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:232) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:378) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:350) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:297) at hudson.plugins.ec2.EC2RetentionStrategy._check(EC2RetentionStrategy.java:99) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:71) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:40) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:70) at hudson.model.Queue._withLock(Queue.java:1286) at hudson.model.Queue.withLock(Queue.java:1169) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:61) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: EC2 instance already terminated: i-1e42a2a7 Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-1e42a2a7 {code}
    via by Jeroen Smolenaers,
  • We use ec2-plugin to start on-demand slaves and terminate them after 30m of idle time. This was working fine, until last week when we noticed a full build queue and an empty list of build executors. On EC2, there actually was a slave running (i-1e42a2a7). Below is the relevant fragment from the log: {code} WARNING: Failed to fetch EC2 instance: i-1e42a2a7 com.amazonaws.AmazonServiceException: The service is unavailable. Please try again shortly. (Service: AmazonEC2; Status Code: 503; Error Code: Unavailable; Re-rw-r---quest ID: ff9e539b-f87f-4143-80b6-e11fbf61378e) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11768) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5899) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:232) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:378) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:350) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:297) at hudson.plugins.ec2.EC2RetentionStrategy._check(EC2RetentionStrategy.java:99) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:71) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:40) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:70) at hudson.model.Queue._withLock(Queue.java:1286) at hudson.model.Queue.withLock(Queue.java:1169) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:61) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: EC2 instance already terminated: i-1e42a2a7 Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-1e42a2a7 {code}
    via by Jeroen Smolenaers,
  • [JIRA] [ec2-plugin] (JENKINS-26493) Use new EC2 API endpoint hostnames
    via by stephan.s...@gmail.com (JIRA),
  • {code} Oct 07, 2014 11:41:27 AM hudson.plugins.ec2.EC2Cloud addProvisionedSlave INFO: Provisioning for AMI ami-abcda06b; Estimated number of total slaves: 241; Estimated number of slaves for ami ami-abcda06b: 30 Launching ami-abcda06b for template aws builder Oct 07, 2014 11:41:27 AM hudson.plugins.ec2.SlaveTemplate provisionOndemand INFO: Launching ami-abcda06b for template aws builder Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-abcda06b]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-1a2bce59]}, {Name: instance.group-id,Values: [sg-e9bb378c, sg-1237d80c]}, {Name: key-name,Values: [jenkins-builder]}, {Name: instance-type,Values: [m3.large]}, {Name: tag:Name,Values: [jenkins-builder.elastic]}, {Name: tag:created_by,Values: [ci.corp.mydomain.com]}, {Name: tag:Service,Values: [jenkins]}, {Name: tag:Team,Values: [releng]}, {Name: tag:Stage,Values: [prod]}, {Name: instance-state-name,Values: [stopped, stopping]}],} Oct 07, 2014 11:41:28 AM hudson.plugins.ec2.EC2Cloud addProvisionedSlave INFO: Provisioning for AMI ami-abcda06b; Estimated number of total slaves: 242; Estimated number of slaves for ami ami-abcda06b: 31 Launching ami-abcda06b for template aws builder Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:14.0 Oct 07, 2014 11:41:28 AM hudson.plugins.ec2.SlaveTemplate provisionOndemand INFO: Launching ami-abcda06b for template aws builder Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:13.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:12.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:11.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:10.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:9.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:8.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:7.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:6.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:5.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:4.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:3.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:2.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:1.0 Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update INFO: Started provisioning aws builder (ami-abcda06b) from ec2-us-east-1 with 1 executors. Remaining excess workload:0.0 Oct 07, 2014 11:41:28 AM hudson.plugins.sauce_ondemand.SauceEnvironmentUtil getBuildWrapper INFO: Could not find SauceOnDemandBuildWrapper on project hudson.model.FreeStyleProject@222a3030[gerrit_cdc] Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update WARNING: Provisioned slave aws builder (ami-abcda06b) failed to launch com.amazonaws.AmazonServiceException: The instance ID 'i-7705eb7d' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.NotFound; Request ID: abcd14da-de6a-40fa-967b-945290c35287) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.createTags(AmazonEC2Client.java:4990) at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:667) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:410) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:284) at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:370) at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:366) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Oct 07, 2014 11:41:28 AM hudson.slaves.NodeProvisioner update WARNING: Provisioned slave aws builder (ami-abcda06b) failed to launch com.amazonaws.AmazonServiceException: The instance ID 'i-9407e99e' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.NotFound; Request ID: abcddcc1-f486-4598-b140-fb1656d24aef) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:385) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:284) at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:370) at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:366) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-abcda06b]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-1a2bce59]}, {Name: instance.group-id,Values: [sg-e9bb378c, sg-1237d80c]}, {Name: key-name,Values: [jenkins-builder]}, {Name: instance-type,Values: [m3.large]}, {Name: tag:Name,Values: [jenkins-builder.elastic]}, {Name: tag:created_by,Values: [ci.corp.mydomain.com]}, {Name: tag:Service,Values: [jenkins]}, {Name: tag:Team,Values: [releng]}, {Name: tag:Stage,Values: [prod]}, {Name: instance-state-name,Values: [stopped, stopping]}],} No existing instance found - created: {InstanceId: i-1abceb45,ImageId: ami-abcda06b,State: {Code: 0,Name: pending},PrivateDnsName: ip-10-32-1-152.us-east-1.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-builder,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m3.large,LaunchTime: Tue Oct 07 11:41:29 PDT 2014,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default},KernelId: aki-98e26fa8,Monitoring: {State: disabled},SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,PrivateIpAddress: 10.32.1.152,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: paravirtual,ClientToken: 58ad84d4-2934-4647-b5e5-d7084db6c33a,Tags: [{Key: Service,Value: jenkins}, {Key: Name,Value: jenkins-builder.elastic}, {Key: Stage,Value: prod}, {Key: Team,Value: releng}, {Key: created_by,Value: ci.corp.mydomain.com}],SecurityGroups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-d4fbeeb1,SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,Description: ,OwnerId: 742630462602,Status: in-use,PrivateIpAddress: 10.32.1.152,PrivateDnsName: ip-10-32-1-152.us-east-1.compute.internal,SourceDestCheck: true,Groups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],Attachment: {AttachmentId: eni-attach-a7b54391,DeviceIndex: 0,Status: attaching,AttachTime: Tue Oct 07 11:41:29 PDT 2014,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.32.1.152,PrivateDnsName: ip-10-32-1-152.us-east-1.compute.internal,Primary: true,}]}],EbsOptimized: false,} Oct 07, 2014 11:41:29 AM hudson.plugins.ec2.SlaveTemplate provisionOndemand INFO: No existing instance found - created: {InstanceId: i-1abceb45,ImageId: ami-abcda06b,State: {Code: 0,Name: pending},PrivateDnsName: ip-10-32-1-152.us-east-1.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-builder,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m3.large,LaunchTime: Tue Oct 07 11:41:29 PDT 2014,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default},KernelId: aki-98e26fa8,Monitoring: {State: disabled},SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,PrivateIpAddress: 10.32.1.152,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: paravirtual,ClientToken: 58ad84d4-2934-4647-b5e5-d7084db6c33a,Tags: [{Key: Service,Value: jenkins}, {Key: Name,Value: jenkins-builder.elastic}, {Key: Stage,Value: prod}, {Key: Team,Value: releng}, {Key: created_by,Value: ci.corp.mydomain.com}],SecurityGroups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-d4fbeeb1,SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,Description: ,OwnerId: 742630462602,Status: in-use,PrivateIpAddress: 10.32.1.152,PrivateDnsName: ip-10-32-1-152.us-east-1.compute.internal,SourceDestCheck: true,Groups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],Attachment: {AttachmentId: eni-attach-a7b54391,DeviceIndex: 0,Status: attaching,AttachTime: Tue Oct 07 11:41:29 PDT 2014,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.32.1.152,PrivateDnsName: ip-10-32-1-152.us-east-1.compute.internal,Primary: true,}]}],EbsOptimized: false,} Oct 07, 2014 11:41:29 AM hudson.plugins.ec2.EC2RetentionStrategy start INFO: Start requested for aws builder (i-1abceb45) No existing instance found - created: {InstanceId: i-2bcdeb44,ImageId: ami-abcda06b,State: {Code: 0,Name: pending},PrivateDnsName: ip-10-32-1-243.us-east-1.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-builder,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m3.large,LaunchTime: Tue Oct 07 11:41:29 PDT 2014,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default},KernelId: aki-98e26fa8,Monitoring: {State: disabled},SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,PrivateIpAddress: 10.32.1.243,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: paravirtual,ClientToken: 2988c7e3-b1ae-4d35-b294-9c1fc36b9d62,Tags: [{Key: Service,Value: jenkins}, {Key: Name,Value: jenkins-builder.elastic}, {Key: Stage,Value: prod}, {Key: Team,Value: releng}, {Key: created_by,Value: ci.corp.mydomain.com}],SecurityGroups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-d6fbeeb3,SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,Description: ,OwnerId: 742630462602,Status: in-use,PrivateIpAddress: 10.32.1.243,PrivateDnsName: ip-10-32-1-243.us-east-1.compute.internal,SourceDestCheck: true,Groups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],Attachment: {AttachmentId: eni-attach-a0b54396,DeviceIndex: 0,Status: attaching,AttachTime: Tue Oct 07 11:41:29 PDT 2014,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.32.1.243,PrivateDnsName: ip-10-32-1-243.us-east-1.compute.internal,Primary: true,}]}],EbsOptimized: false,} Oct 07, 2014 11:41:29 AM hudson.plugins.ec2.SlaveTemplate provisionOndemand INFO: No existing instance found - created: {InstanceId: i-2bcdeb44,ImageId: ami-abcda06b,State: {Code: 0,Name: pending},PrivateDnsName: ip-10-32-1-243.us-east-1.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-builder,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m3.large,LaunchTime: Tue Oct 07 11:41:29 PDT 2014,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default},KernelId: aki-98e26fa8,Monitoring: {State: disabled},SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,PrivateIpAddress: 10.32.1.243,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: paravirtual,ClientToken: 2988c7e3-b1ae-4d35-b294-9c1fc36b9d62,Tags: [{Key: Service,Value: jenkins}, {Key: Name,Value: jenkins-builder.elastic}, {Key: Stage,Value: prod}, {Key: Team,Value: releng}, {Key: created_by,Value: ci.corp.mydomain.com}],SecurityGroups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-d6fbeeb3,SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,Description: ,OwnerId: 742630462602,Status: in-use,PrivateIpAddress: 10.32.1.243,PrivateDnsName: ip-10-32-1-243.us-east-1.compute.internal,SourceDestCheck: true,Groups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],Attachment: {AttachmentId: eni-attach-a0b54396,DeviceIndex: 0,Status: attaching,AttachTime: Tue Oct 07 11:41:29 PDT 2014,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.32.1.243,PrivateDnsName: ip-10-32-1-243.us-east-1.compute.internal,Primary: true,}]}],EbsOptimized: false,} Oct 07, 2014 11:41:29 AM hudson.plugins.ec2.EC2RetentionStrategy start INFO: Start requested for aws builder (i-2bcdeb44) No existing instance found - created: {InstanceId: i-a709e7ad,ImageId: ami-abcda06b,State: {Code: 0,Name: pending},PrivateDnsName: ip-10-32-1-22.us-east-1.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-builder,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m3.large,LaunchTime: Tue Oct 07 11:41:30 PDT 2014,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default},KernelId: aki-98e26fa8,Monitoring: {State: disabled},SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,PrivateIpAddress: 10.32.1.22,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: paravirtual,ClientToken: 171b131a-7067-4ac8-b106-89e1400304b1,Tags: [{Key: Service,Value: jenkins}, {Key: Name,Value: jenkins-builder.elastic}, {Key: Stage,Value: prod}, {Key: Team,Value: releng}, {Key: created_by,Value: ci.corp.mydomain.com}],SecurityGroups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-d1fbeeb4,SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,Description: ,OwnerId: 742630462602,Status: in-use,PrivateIpAddress: 10.32.1.22,PrivateDnsName: ip-10-32-1-22.us-east-1.compute.internal,SourceDestCheck: true,Groups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],Attachment: {AttachmentId: eni-attach-3ab3450c,DeviceIndex: 0,Status: attaching,AttachTime: Tue Oct 07 11:41:30 PDT 2014,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.32.1.22,PrivateDnsName: ip-10-32-1-22.us-east-1.compute.internal,Primary: true,}]}],EbsOptimized: false,} Oct 07, 2014 11:41:30 AM hudson.plugins.ec2.SlaveTemplate provisionOndemand INFO: No existing instance found - created: {InstanceId: i-a709e7ad,ImageId: ami-abcda06b,State: {Code: 0,Name: pending},PrivateDnsName: ip-10-32-1-22.us-east-1.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-builder,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m3.large,LaunchTime: Tue Oct 07 11:41:30 PDT 2014,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default},KernelId: aki-98e26fa8,Monitoring: {State: disabled},SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,PrivateIpAddress: 10.32.1.22,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: paravirtual,ClientToken: 171b131a-7067-4ac8-b106-89e1400304b1,Tags: [{Key: Service,Value: jenkins}, {Key: Name,Value: jenkins-builder.elastic}, {Key: Stage,Value: prod}, {Key: Team,Value: releng}, {Key: created_by,Value: ci.corp.mydomain.com}],SecurityGroups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-d1fbeeb4,SubnetId: subnet-1a2bce59,VpcId: vpc-26a44a43,Description: ,OwnerId: 742630462602,Status: in-use,PrivateIpAddress: 10.32.1.22,PrivateDnsName: ip-10-32-1-22.us-east-1.compute.internal,SourceDestCheck: true,Groups: [{GroupName: corp,GroupId: sg-e9bb378c}, {GroupName: jenkins,GroupId: sg-1237d80c}],Attachment: {AttachmentId: eni-attach-3ab3450c,DeviceIndex: 0,Status: attaching,AttachTime: Tue Oct 07 11:41:30 PDT 2014,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.32.1.22,PrivateDnsName: ip-10-32-1-22.us-east-1.compute.internal,Primary: true,}]}],EbsOptimized: false,} O {code} Not sure this is related or not too: {code} Oct 06, 2014 4:28:24 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Failed to count the # of live instances on EC2 com.amazonaws.AmazonServiceException: Request limit exceeded. (Service: AmazonEC2; Status Code: 503; Error Code: RequestLimitExceeded; Request ID: 12c06f12-abcd-4c56-bae5-51ff7fa83423) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:212) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:272) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:361) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) {code}
    via by Sam Xiao,
  • After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label. A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem. {code}Started EC2 alive slaves monitor Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished EC2 alive slaves monitor. 0 ms Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision Excess workload after pending Spot instances: 1 Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision Failed to count the # of live instances on EC2 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745){code} I also don't understand the log statement {{Excess workload after pending Spot instances: 1}} as I have not ticked the "Use Spot instance" tick box. In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".
    via by Ximon Eighteen,
    • com.amazonaws.AmazonServiceException: The service is unavailable. Please try again shortly. (Service: AmazonEC2; Status Code: 503; Error Code: Unavailable; Re-rw-r---quest ID: ff9e539b-f87f-4143-80b6-e11fbf61378e) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11768) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5899) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:232) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:378) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:350) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:297) at hudson.plugins.ec2.EC2RetentionStrategy._check(EC2RetentionStrategy.java:99) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:71) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:40) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:70) at hudson.model.Queue._withLock(Queue.java:1286) at hudson.model.Queue.withLock(Queue.java:1169) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:61) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    Users with the same issue

    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    43 more bugmates