java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187

Jenkins JIRA | Steve Roth | 3 years ago
  1. 0

    Summary: ======== When Jenkins results are stored on a NAS location, we are seeing very frequent errors (100x/day) when log rotation runs after a job completes. These appear to be a very small (4-second or less) race condition when using NAS storage. The net effect is that log rotation does not appear to work properly, at least from the Jenkins perspective, resulting in increasing build records over time. The typical symptom, as seen in the Jenkins console, is: {noformat} Mar 15, 2014 10:38:32 PM hudson.model.Run execute SEVERE: Failed to rotate log java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187 at hudson.Util.deleteFile(Util.java:254) at hudson.Util.deleteRecursive(Util.java:301) at hudson.Util.deleteContentsRecursive(Util.java:203) at hudson.Util.deleteRecursive(Util.java:300) at hudson.model.Run.delete(Run.java:1452) at hudson.tasks.LogRotator.perform(LogRotator.java:124) at hudson.model.Job.logRotate(Job.java:440) at hudson.model.Run.execute(Run.java:1739) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) {noformat} More Details: ============== We have a good-sized Jenkins install, with several hundred jobs, and due to the archiving size requirements and large # of jobs, we are archiving results to a NAS location. The Jenkins Build Record Root Directory option is configured to /zzz/jenkinsarchive/${ITEM_FULLNAME}/builds Where /zzz/jenkinsarchive is a symlink to a mount-point location, such as /mnt/foo.com/export/foo/bar/jenkinsarchive So records are written to /zzz/jenkinsarchive/SomeJob/builds/... which really evaluates to /mnt/foo.com/export/foo/bar/jenkinsarchive/SomeJob/builds/... My understanding of .nfs lock files is that they appear on the NAS when a NAS file is unlinked but is still open by a NFS client. (presumably in this case, Jenkins). Using lsof -n -P -N -u someUser -a I can see that normally we have 2 files in-use on the NAS -- one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/log and one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/timestamper/timestamps but for very short periods of time, when log rotation runs, I see a files at a path like jenkinsarchive/SomeJob/builds/.2014-03-15_19-19-58/.nfs000000000009e0cc000003c4 exist. It appears to only exist for a couple seconds. So it seems that something in Jenkins is keeping access open to these files, which then (per NFS semantics) is preventing deletion at the time of the stack trace above.

    Jenkins JIRA | 3 years ago | Steve Roth
    java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187
  2. 0

    Summary: ======== When Jenkins results are stored on a NAS location, we are seeing very frequent errors (100x/day) when log rotation runs after a job completes. These appear to be a very small (4-second or less) race condition when using NAS storage. The net effect is that log rotation does not appear to work properly, at least from the Jenkins perspective, resulting in increasing build records over time. The typical symptom, as seen in the Jenkins console, is: {noformat} Mar 15, 2014 10:38:32 PM hudson.model.Run execute SEVERE: Failed to rotate log java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187 at hudson.Util.deleteFile(Util.java:254) at hudson.Util.deleteRecursive(Util.java:301) at hudson.Util.deleteContentsRecursive(Util.java:203) at hudson.Util.deleteRecursive(Util.java:300) at hudson.model.Run.delete(Run.java:1452) at hudson.tasks.LogRotator.perform(LogRotator.java:124) at hudson.model.Job.logRotate(Job.java:440) at hudson.model.Run.execute(Run.java:1739) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) {noformat} More Details: ============== We have a good-sized Jenkins install, with several hundred jobs, and due to the archiving size requirements and large # of jobs, we are archiving results to a NAS location. The Jenkins Build Record Root Directory option is configured to /zzz/jenkinsarchive/${ITEM_FULLNAME}/builds Where /zzz/jenkinsarchive is a symlink to a mount-point location, such as /mnt/foo.com/export/foo/bar/jenkinsarchive So records are written to /zzz/jenkinsarchive/SomeJob/builds/... which really evaluates to /mnt/foo.com/export/foo/bar/jenkinsarchive/SomeJob/builds/... My understanding of .nfs lock files is that they appear on the NAS when a NAS file is unlinked but is still open by a NFS client. (presumably in this case, Jenkins). Using lsof -n -P -N -u someUser -a I can see that normally we have 2 files in-use on the NAS -- one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/log and one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/timestamper/timestamps but for very short periods of time, when log rotation runs, I see a files at a path like jenkinsarchive/SomeJob/builds/.2014-03-15_19-19-58/.nfs000000000009e0cc000003c4 exist. It appears to only exist for a couple seconds. So it seems that something in Jenkins is keeping access open to these files, which then (per NFS semantics) is preventing deletion at the time of the stack trace above.

    Jenkins JIRA | 3 years ago | Steve Roth
    java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187
  3. 0

    Version 0.8 - incompatibility with Docker 1.4.0 - 'Failed to mkdirs'

    GitHub | 2 years ago | hcguersoy
    java.io.IOException: Failed to install http://archive.apache.org/dist/maven/binaries/apache-maven-3.2.1-bin.zip to /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/3.2.1
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    ClearCase Plugin - hudson - Hudson Wiki

    hudson-ci.org | 7 months ago
    java.io.IOException: Failed to mkdirs: V:\XXXXXXX
  6. 0

    Only when user not logged in to the Ubuntu slave: java.io.IOException: Failed to mkdirs

    Google Groups | 4 years ago | Pasi
    java.io.IOException: Failed to mkdirs: /home/ouruser/jenkins/workspace/network_test_summary

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187

      at hudson.Util.deleteFile()
    2. Hudson
      Executor.run
      1. hudson.Util.deleteFile(Util.java:254)
      2. hudson.Util.deleteRecursive(Util.java:301)
      3. hudson.Util.deleteContentsRecursive(Util.java:203)
      4. hudson.Util.deleteRecursive(Util.java:300)
      5. hudson.model.Run.delete(Run.java:1452)
      6. hudson.tasks.LogRotator.perform(LogRotator.java:124)
      7. hudson.model.Job.logRotate(Job.java:440)
      8. hudson.model.Run.execute(Run.java:1739)
      9. hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      10. hudson.model.ResourceController.execute(ResourceController.java:88)
      11. hudson.model.Executor.run(Executor.java:231)
      11 frames