com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • When using docker slaves on Amazon EC2 Container Service, the slave tasks aren't being cleaned up properly. It says the referenced task was not found but when looking at the AWS Console, the tasks are shown with the correct UUIDs. The following stack trace occurs when a project completes building on a docker slave: {noformat} INFO: Slave ecs-cloud-93f834832c1 - Slave Task Started : arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:49 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #13 from /10.0.0.219:35968 Feb 11, 2016 2:29:51 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave ecs-cloud-93f834832c1 (ecs task arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) connected Feb 11, 2016 2:29:52 PM hudson.model.Run execute INFO: ecs-test #3 main build action completed: SUCCESS Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask INFO: Delete ECS Slave task: arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask SEVERE: Couldn't stop task arn arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx caught exception: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2220) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1807) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:212) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:145) at hudson.model.Executor.finish1(Executor.java:460) at hudson.model.Executor.run(Executor.java:430) Feb 11, 2016 2:29:52 PM hudson.remoting.AbstractByteArrayCommandTransport$1 handle WARNING: Failed to construct Command java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61) at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:594) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat}
    via by Dieter De Meyer,
  • When using docker slaves on Amazon EC2 Container Service, the slave tasks aren't being cleaned up properly. It says the referenced task was not found but when looking at the AWS Console, the tasks are shown with the correct UUIDs. The following stack trace occurs when a project completes building on a docker slave: {noformat} INFO: Slave ecs-cloud-93f834832c1 - Slave Task Started : arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:49 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #13 from /10.0.0.219:35968 Feb 11, 2016 2:29:51 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave ecs-cloud-93f834832c1 (ecs task arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) connected Feb 11, 2016 2:29:52 PM hudson.model.Run execute INFO: ecs-test #3 main build action completed: SUCCESS Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask INFO: Delete ECS Slave task: arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask SEVERE: Couldn't stop task arn arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx caught exception: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2220) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1807) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:212) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:145) at hudson.model.Executor.finish1(Executor.java:460) at hudson.model.Executor.run(Executor.java:430) Feb 11, 2016 2:29:52 PM hudson.remoting.AbstractByteArrayCommandTransport$1 handle WARNING: Failed to construct Command java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61) at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:594) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat}
    via by Dieter De Meyer,
  • On every Jenkins restart (or start) the ECS plugin throws a "The reference task was not found" exception which prevents Jenkins from starting correctly. I've also noticed that there are occasions when this happens when you are attempting to apply or save within the Jenkins configuration which bubbles up to the UI & stops you from saving. The full stack trace is here: {code} hudson.util.HudsonFailedToLoad: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd) at hudson.WebAppMain$3.run(WebAppMain.java:237) Caused by: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1181) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:766) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:306) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2199) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1874) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:205) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:58) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:42) at hudson.slaves.SlaveComputer$4.run(SlaveComputer.java:717) at hudson.model.Queue._withLock(Queue.java:1346) at hudson.model.Queue.withLock(Queue.java:1229) at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:714) at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:118) at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:44) at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:186) at hudson.model.Queue._withLock(Queue.java:1346) at hudson.model.Queue.withLock(Queue.java:1229) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:169) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1247) at jenkins.model.Jenkins.<init>(Jenkins.java:844) at hudson.model.Hudson.<init>(Hudson.java:83) at hudson.model.Hudson.<init>(Hudson.java:79) at hudson.WebAppMain$3.run(WebAppMain.java:225) {code} When you have containers / tasks which execute correctly & aren't spawning a lot of builds then it happens less, but when you've got issues with the `jenkins-slave` entrypoint coming up correctly of the server rejecting the JNLP agent (because of a key issue or something) then it happens a lot & basically stops you from using the configuration UI correctly. A workaround on Jenkins startup is to delete /var/lib/jenkins/plugins/amazon-ecs & then the startup can happen normally I'm thinking of putting a try / catch around com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:205) but am not sure whether this is appropriate (it'll certainly stop the issues above though) Thoughts before I make a PR?
    via by Lee Webb,
  • On every Jenkins restart (or start) the ECS plugin throws a "The reference task was not found" exception which prevents Jenkins from starting correctly. I've also noticed that there are occasions when this happens when you are attempting to apply or save within the Jenkins configuration which bubbles up to the UI & stops you from saving. The full stack trace is here: {code} hudson.util.HudsonFailedToLoad: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd) at hudson.WebAppMain$3.run(WebAppMain.java:237) Caused by: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1181) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:766) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:306) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2199) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1874) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:205) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:58) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:42) at hudson.slaves.SlaveComputer$4.run(SlaveComputer.java:717) at hudson.model.Queue._withLock(Queue.java:1346) at hudson.model.Queue.withLock(Queue.java:1229) at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:714) at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:118) at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:44) at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:186) at hudson.model.Queue._withLock(Queue.java:1346) at hudson.model.Queue.withLock(Queue.java:1229) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:169) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1247) at jenkins.model.Jenkins.<init>(Jenkins.java:844) at hudson.model.Hudson.<init>(Hudson.java:83) at hudson.model.Hudson.<init>(Hudson.java:79) at hudson.WebAppMain$3.run(WebAppMain.java:225) {code} When you have containers / tasks which execute correctly & aren't spawning a lot of builds then it happens less, but when you've got issues with the `jenkins-slave` entrypoint coming up correctly of the server rejecting the JNLP agent (because of a key issue or something) then it happens a lot & basically stops you from using the configuration UI correctly. A workaround on Jenkins startup is to delete /var/lib/jenkins/plugins/amazon-ecs & then the startup can happen normally I'm thinking of putting a try / catch around com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:205) but am not sure whether this is appropriate (it'll certainly stop the issues above though) Thoughts before I make a PR?
    via by Lee Webb,
    • com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2220) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1807) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:212) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:145) at hudson.model.Executor.finish1(Executor.java:460) at hudson.model.Executor.run(Executor.java:430)
    No Bugmate found.