java.io.IOException: Unable to delete '/var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

jenkins-ci.org | 4 months ago
  1. 0

    [JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks

    Google Groups | 4 months ago | ev...@borgstrom.ca (JIRA)
    java.io.IOException: Unable to delete '/var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
  2. 0

    [JENKINS-37293] ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks - Jenkins JIRA

    jenkins-ci.org | 4 months ago
    java.io.IOException: Unable to delete '/var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
  3. 0

    We have our Jenkins master running on a NFS 4.1 mount (AWS EFS). When a job completes we see the following in the logs: {noformat} Aug 09, 2016 4:21:09 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSService deleteTask Delete ECS Slave task: arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task/80b3bc38-774c-4bbe-9e79-f9db126a2b9b Aug 09, 2016 4:21:09 PM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor jenkins-ecs-d6d51f89f024 for Free Disk Space java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Request$1.get(Request.java:282) at hudson.remoting.Request$1.get(Request.java:207) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Request.abort(Request.java:303) at hudson.remoting.Channel.terminate(Channel.java:847) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1080) at hudson.remoting.Channel$1.handle(Channel.java:501) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83) Caused by: hudson.remoting.Channel$OrderlyShutdown ... 3 more Caused by: Command close created at at hudson.remoting.Command.<init>(Command.java:62) at hudson.remoting.Command.<init>(Command.java:47) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:822) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:822) at hudson.remoting.Channel.close(Channel.java:867) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:825) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1106) Aug 09, 2016 4:21:09 PM WARNING hudson.slaves.SlaveComputer kill Unable to delete agent logs java.io.IOException: Unable to delete '/var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteRecursive(Util.java:355) at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:670) at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:86) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:205) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1338) at jenkins.model.Nodes$4.run(Nodes.java:219) at hudson.model.Queue._withLock(Queue.java:1320) at hudson.model.Queue.withLock(Queue.java:1197) at jenkins.model.Nodes.removeNode(Nodes.java:210) at jenkins.model.Jenkins.removeNode(Jenkins.java:1860) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:70) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62) at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140) at hudson.model.Executor.finish1(Executor.java:457) at hudson.model.Executor.run(Executor.java:430) Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024/.nfs9e9ba4fd7c349fa200000003: Device or resource busy at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1165) at hudson.Util.tryOnceDeleteFile(Util.java:296) at hudson.Util.tryOnceDeleteRecursive(Util.java:373) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:392) at hudson.Util.tryOnceDeleteRecursive(Util.java:372) at hudson.Util.deleteRecursive(Util.java:350) ... 15 more {noformat} Looking at the Jenkins master I do indeed see the log file still open: {noformat} eborgstrom@devops-jenkins-7d98bfe3:/var/lib/jenkins$ sudo lsof +D /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 26119 jenkins 486w REG 0,25 22 11428909888000270242 /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024/.nfs9e9ba4fd7c349fa200000003 eborgstrom@devops-jenkins-7d98bfe3:/var/lib/jenkins$ find /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024 -ls 414054371676357616 4 drwxr-xr-x 2 jenkins jenkins 4096 Aug 9 23:21 /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024 11428909888000270242 4 -rw-r--r-- 1 jenkins jenkins 22 Aug 9 23:21 /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024/.nfs9e9ba4fd7c349fa200000003 {noformat} This means agents remain around and need to be cleaned up manually. The logs should be fully closed before the slave is deleted.

    Jenkins JIRA | 4 months ago | Evan Borgstrom
    java.io.IOException: Unable to delete '/var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Java file system exception when deleting file with NIO Files.delete()

    Stack Overflow | 2 years ago
    java.nio.file.FileSystemException: C:\Users\abcd.zip: The process cannot access the file because it is being used by another process.
  6. 0

    Report an error if a temporary file is locked by another process

    GitHub | 11 months ago | fflewddur
    java.nio.file.FileSystemException: D:\Users\Todd\Videos\Tivo Exports\Masterpiece Mystery! - S07E22 - Grantchester, Episode 3.cut.ts: The process cannot access the file because it is being used by another process.

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.nio.file.FileSystemException

      /var/lib/jenkins/logs/slaves/jenkins-ecs-d6d51f89f024/.nfs9e9ba4fd7c349fa200000003: Device or resource busy

      at sun.nio.fs.UnixException.translateToIOException()
    2. Java RT
      Files.deleteIfExists
      1. sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
      2. sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      3. sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      4. sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
      5. sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
      6. java.nio.file.Files.deleteIfExists(Files.java:1165)
      6 frames
    3. Hudson
      AbstractCIBase.updateComputerList
      1. hudson.Util.tryOnceDeleteFile(Util.java:296)
      2. hudson.Util.tryOnceDeleteRecursive(Util.java:373)
      3. hudson.Util.tryOnceDeleteContentsRecursive(Util.java:392)
      4. hudson.Util.tryOnceDeleteRecursive(Util.java:372)
      5. hudson.Util.deleteRecursive(Util.java:350)
      6. hudson.slaves.SlaveComputer.kill(SlaveComputer.java:670)
      7. hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:86)
      8. hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:205)
      8 frames
    4. jenkins.model
      Nodes$4.run
      1. jenkins.model.Jenkins.updateComputerList(Jenkins.java:1338)
      2. jenkins.model.Nodes$4.run(Nodes.java:219)
      2 frames
    5. Hudson
      Queue.withLock
      1. hudson.model.Queue._withLock(Queue.java:1320)
      2. hudson.model.Queue.withLock(Queue.java:1197)
      2 frames
    6. jenkins.model
      Jenkins.removeNode
      1. jenkins.model.Nodes.removeNode(Nodes.java:210)
      2. jenkins.model.Jenkins.removeNode(Jenkins.java:1860)
      2 frames
    7. Hudson
      AbstractCloudSlave.terminate
      1. hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:70)
      1 frame
    8. com.cloudbees.jenkins
      ECSComputer.taskCompleted
      1. com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:62)
      2. com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:47)
      2 frames
    9. Hudson
      Executor.run
      1. hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140)
      2. hudson.model.Executor.finish1(Executor.java:457)
      3. hudson.model.Executor.run(Executor.java:430)
      3 frames