com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

Jenkins JIRA | Dieter De Meyer | 10 months ago
  1. 0

    [JIRA] [amazon-ecs-plugin] (JENKINS-32895) "The referenced task was not found" prevents docker slaves from being deleted

    Google Groups | 10 months ago | didi357@java.net (JIRA)
    com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
  2. 0

    When using docker slaves on Amazon EC2 Container Service, the slave tasks aren't being cleaned up properly. It says the referenced task was not found but when looking at the AWS Console, the tasks are shown with the correct UUIDs. The following stack trace occurs when a project completes building on a docker slave: {noformat} INFO: Slave ecs-cloud-93f834832c1 - Slave Task Started : arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:49 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #13 from /10.0.0.219:35968 Feb 11, 2016 2:29:51 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave ecs-cloud-93f834832c1 (ecs task arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) connected Feb 11, 2016 2:29:52 PM hudson.model.Run execute INFO: ecs-test #3 main build action completed: SUCCESS Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask INFO: Delete ECS Slave task: arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask SEVERE: Couldn't stop task arn arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx caught exception: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2220) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1807) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:212) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:145) at hudson.model.Executor.finish1(Executor.java:460) at hudson.model.Executor.run(Executor.java:430) Feb 11, 2016 2:29:52 PM hudson.remoting.AbstractByteArrayCommandTransport$1 handle WARNING: Failed to construct Command java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61) at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:594) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat}

    Jenkins JIRA | 10 months ago | Dieter De Meyer
    com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
  3. 0

    When using docker slaves on Amazon EC2 Container Service, the slave tasks aren't being cleaned up properly. It says the referenced task was not found but when looking at the AWS Console, the tasks are shown with the correct UUIDs. The following stack trace occurs when a project completes building on a docker slave: {noformat} INFO: Slave ecs-cloud-93f834832c1 - Slave Task Started : arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:49 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #13 from /10.0.0.219:35968 Feb 11, 2016 2:29:51 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave ecs-cloud-93f834832c1 (ecs task arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) connected Feb 11, 2016 2:29:52 PM hudson.model.Run execute INFO: ecs-test #3 main build action completed: SUCCESS Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask INFO: Delete ECS Slave task: arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Feb 11, 2016 2:29:52 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud deleteTask SEVERE: Couldn't stop task arn arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx caught exception: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2220) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1807) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:212) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:145) at hudson.model.Executor.finish1(Executor.java:460) at hudson.model.Executor.run(Executor.java:430) Feb 11, 2016 2:29:52 PM hudson.remoting.AbstractByteArrayCommandTransport$1 handle WARNING: Failed to construct Command java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61) at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:594) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat}

    Jenkins JIRA | 10 months ago | Dieter De Meyer
    com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [JIRA] [amazon-ecs-plugin] (JENKINS-32321) "The reference task was not found" prevents Jenkins from starting

    Google Groups | 11 months ago | nullify005@gmail.com (JIRA)
    hudson.util.HudsonFailedToLoad: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd)
  6. 0

    On every Jenkins restart (or start) the ECS plugin throws a "The reference task was not found" exception which prevents Jenkins from starting correctly. I've also noticed that there are occasions when this happens when you are attempting to apply or save within the Jenkins configuration which bubbles up to the UI & stops you from saving. The full stack trace is here: {code} hudson.util.HudsonFailedToLoad: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd) at hudson.WebAppMain$3.run(WebAppMain.java:237) Caused by: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1181) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:766) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:306) at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2199) at com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1874) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:205) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:58) at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:42) at hudson.slaves.SlaveComputer$4.run(SlaveComputer.java:717) at hudson.model.Queue._withLock(Queue.java:1346) at hudson.model.Queue.withLock(Queue.java:1229) at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:714) at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:118) at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:44) at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:186) at hudson.model.Queue._withLock(Queue.java:1346) at hudson.model.Queue.withLock(Queue.java:1229) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:169) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1247) at jenkins.model.Jenkins.<init>(Jenkins.java:844) at hudson.model.Hudson.<init>(Hudson.java:83) at hudson.model.Hudson.<init>(Hudson.java:79) at hudson.WebAppMain$3.run(WebAppMain.java:225) {code} When you have containers / tasks which execute correctly & aren't spawning a lot of builds then it happens less, but when you've got issues with the `jenkins-slave` entrypoint coming up correctly of the server rejecting the JNLP agent (because of a key issue or something) then it happens a lot & basically stops you from using the configuration UI correctly. A workaround on Jenkins startup is to delete /var/lib/jenkins/plugins/amazon-ecs & then the startup can happen normally I'm thinking of putting a try / catch around com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:205) but am not sure whether this is appropriate (it'll certainly stop the issues above though) Thoughts before I make a PR?

    Jenkins JIRA | 11 months ago | Lee Webb
    hudson.util.HudsonFailedToLoad: com.amazonaws.services.ecs.model.ClientException: The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 041ee786-b4c1-11e5-a864-d7bdaaa4b5cd)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.amazonaws.services.ecs.model.ClientException

      The referenced task was not found. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

      at com.amazonaws.http.AmazonHttpClient.handleErrorResponse()
    2. AWS SDK for Java - Core
      AmazonHttpClient.execute
      1. com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275)
      2. com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873)
      3. com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576)
      4. com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362)
      5. com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328)
      6. com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307)
      6 frames
    3. AWS Java SDK for the Amazon EC2 Container Service
      AmazonECSClient.stopTask
      1. com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:2220)
      2. com.amazonaws.services.ecs.AmazonECSClient.stopTask(AmazonECSClient.java:1807)
      2 frames
    4. com.cloudbees.jenkins
      ECSSlave._terminate
      1. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.deleteTask(ECSCloud.java:212)
      2. com.cloudbees.jenkins.plugins.amazonecs.ECSSlave._terminate(ECSSlave.java:90)
      2 frames
    5. Hudson
      AbstractCloudSlave.terminate
      1. hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67)
      1 frame
    6. com.cloudbees.jenkins
      ECSComputer.taskCompleted
      1. com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62)
      2. com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47)
      2 frames
    7. Hudson
      Executor.run
      1. hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:145)
      2. hudson.model.Executor.finish1(Executor.java:460)
      3. hudson.model.Executor.run(Executor.java:430)
      3 frames