java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Summary: ======== When Jenkins results are stored on a NAS location, we are seeing very frequent errors (100x/day) when log rotation runs after a job completes. These appear to be a very small (4-second or less) race condition when using NAS storage. The net effect is that log rotation does not appear to work properly, at least from the Jenkins perspective, resulting in increasing build records over time. The typical symptom, as seen in the Jenkins console, is: {noformat} Mar 15, 2014 10:38:32 PM hudson.model.Run execute SEVERE: Failed to rotate log java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187 at hudson.Util.deleteFile(Util.java:254) at hudson.Util.deleteRecursive(Util.java:301) at hudson.Util.deleteContentsRecursive(Util.java:203) at hudson.Util.deleteRecursive(Util.java:300) at hudson.model.Run.delete(Run.java:1452) at hudson.tasks.LogRotator.perform(LogRotator.java:124) at hudson.model.Job.logRotate(Job.java:440) at hudson.model.Run.execute(Run.java:1739) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) {noformat} More Details: ============== We have a good-sized Jenkins install, with several hundred jobs, and due to the archiving size requirements and large # of jobs, we are archiving results to a NAS location. The Jenkins Build Record Root Directory option is configured to /zzz/jenkinsarchive/${ITEM_FULLNAME}/builds Where /zzz/jenkinsarchive is a symlink to a mount-point location, such as /mnt/foo.com/export/foo/bar/jenkinsarchive So records are written to /zzz/jenkinsarchive/SomeJob/builds/... which really evaluates to /mnt/foo.com/export/foo/bar/jenkinsarchive/SomeJob/builds/... My understanding of .nfs lock files is that they appear on the NAS when a NAS file is unlinked but is still open by a NFS client. (presumably in this case, Jenkins). Using lsof -n -P -N -u someUser -a I can see that normally we have 2 files in-use on the NAS -- one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/log and one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/timestamper/timestamps but for very short periods of time, when log rotation runs, I see a files at a path like jenkinsarchive/SomeJob/builds/.2014-03-15_19-19-58/.nfs000000000009e0cc000003c4 exist. It appears to only exist for a couple seconds. So it seems that something in Jenkins is keeping access open to these files, which then (per NFS semantics) is preventing deletion at the time of the stack trace above.
    via by Steve Roth,
  • Summary: ======== When Jenkins results are stored on a NAS location, we are seeing very frequent errors (100x/day) when log rotation runs after a job completes. These appear to be a very small (4-second or less) race condition when using NAS storage. The net effect is that log rotation does not appear to work properly, at least from the Jenkins perspective, resulting in increasing build records over time. The typical symptom, as seen in the Jenkins console, is: {noformat} Mar 15, 2014 10:38:32 PM hudson.model.Run execute SEVERE: Failed to rotate log java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187 at hudson.Util.deleteFile(Util.java:254) at hudson.Util.deleteRecursive(Util.java:301) at hudson.Util.deleteContentsRecursive(Util.java:203) at hudson.Util.deleteRecursive(Util.java:300) at hudson.model.Run.delete(Run.java:1452) at hudson.tasks.LogRotator.perform(LogRotator.java:124) at hudson.model.Job.logRotate(Job.java:440) at hudson.model.Run.execute(Run.java:1739) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) {noformat} More Details: ============== We have a good-sized Jenkins install, with several hundred jobs, and due to the archiving size requirements and large # of jobs, we are archiving results to a NAS location. The Jenkins Build Record Root Directory option is configured to /zzz/jenkinsarchive/${ITEM_FULLNAME}/builds Where /zzz/jenkinsarchive is a symlink to a mount-point location, such as /mnt/foo.com/export/foo/bar/jenkinsarchive So records are written to /zzz/jenkinsarchive/SomeJob/builds/... which really evaluates to /mnt/foo.com/export/foo/bar/jenkinsarchive/SomeJob/builds/... My understanding of .nfs lock files is that they appear on the NAS when a NAS file is unlinked but is still open by a NFS client. (presumably in this case, Jenkins). Using lsof -n -P -N -u someUser -a I can see that normally we have 2 files in-use on the NAS -- one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/log and one at jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/timestamper/timestamps but for very short periods of time, when log rotation runs, I see a files at a path like jenkinsarchive/SomeJob/builds/.2014-03-15_19-19-58/.nfs000000000009e0cc000003c4 exist. It appears to only exist for a couple seconds. So it seems that something in Jenkins is keeping access open to these files, which then (per NFS semantics) is preventing deletion at the time of the stack trace above.
    via by Steve Roth,
  • ClearCase Plugin - hudson - Hudson Wiki
    via by Unknown author,
  • Jenkins - Automatic Java installation problem
    via by Mahadi Hasan,
  • ClearCase Plugin - Jenkins - Jenkins Wiki
    via by Unknown author,
  • Hi, I have a problem - which is almost the same problem of this issue about mkdir/open files on remote disk. Currently I'm using Jenkins ver. 1.467 and 1.470 with Windows XP 32bit. When I want to checkout source code to remote disk whether it is on MAC OS or Linux, I'll get mkdir fail. I'm very sure java.exe, jenkins.exe, git.exe, p4.exe are executing as my account in Windows XP. And this account has right to read/write/mkdir/delete the files and directories in the remote disk. It seems this problem exists a very long time. Started by user anonymous Building in workspace Z:\PC12010025 java.io.IOException: Failed to mkdirs: Z:\PC12010025 at hudson.FilePath.mkdirs(FilePath.java:901) at hudson.model.AbstractProject.checkout(AbstractProject.java:1216) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:587) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:476) at hudson.model.Run.run(Run.java:1438) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:239) Finished: FAILURE There are other users encountered this problem like http://serverfault.com/questions/308774/how-to-make-hudson-write-to-r... And the relative source code in Jenkins is: /** * Creates this directory. */ public void mkdirs() throws IOException, InterruptedException { if(!act(new FileCallable<Boolean>() { public Boolean invoke(File f, VirtualChannel channel) throws IOException, InterruptedException { if(f.mkdirs() || f.exists()) return true; // OK // following Ant <mkdir> task to avoid possible race condition. Thread.sleep(10); return f.mkdirs() || f.exists(); } })) throw new IOException("Failed to mkdirs: "+remote); }
    via by Macpaul Lin,
  • I want the Jenkins master to build a small project on a slave. Slave and Master run Suse Linux. The same user (hans) which runs Jenkins is available on both machines and has rw permissions on the slave's remote working directory (/var/jenkins). The slave agent is launched on the slave ('ttvm3'). Building the project locally works. When I configure the project to be build on the slave, it fails. The console output on the master is: {quote} Started by user anonymous Building on master in workspace /home/hans/.jenkins/jobs/testproject/workspace Checkout:workspace / /home/hans/.jenkins/jobs/testproject/workspace - hudson.remoting.LocalChannel@11b1e39 Using strategy: Default Last Built Revision: Revision 97957e558fed7d0b116950e09dec1c248d1d0b54 (origin/HEAD, origin/master) Checkout:workspace / /home/hans/.jenkins/jobs/testproject/workspace - hudson.remoting.LocalChannel@11b1e39 Fetching changes from 1 remote Git repository Fetching upstream changes from <<blablabla>> Commencing build of Revision 69e242182e55e57b56c836a9c3a34f0232e5d56c (origin/master) Checking out Revision 69e242182e55e57b56c836a9c3a34f0232e5d56c (origin/master) Triggering ttvm3 ttvm3 completed with result FAILURE Finished: FAILURE {quote} The output on the slave for this build is: {quote} Started by upstream project "testproject" build number 29 Building remotely on ttvm3 in workspace /ttvm3 java.io.IOException: Failed to mkdirs: /ttvm3 at hudson.FilePath.mkdirs(FilePath.java:847) at hudson.model.AbstractProject.checkout(AbstractProject.java:1193) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:576) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:465) at hudson.model.Run.run(Run.java:1409) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:238) Finished: FAILURE {quote} It seems as if Jenkins wants to create a directory called "ttvm3" (which is the name of the slave) in the root directory of the slave. When I (just for the fun of it) create the directory /ttvm3 on the slave and build the project again, the output for this build on the slave is: {quote} Started by upstream project "testproject" build number 30 Building remotely on ttvm3 in workspace /ttvm3 Checkout:ttvm3 / /ttvm3 - hudson.remoting.Channel@c4931d:ttvm3 Using strategy: Default Checkout:ttvm3 / /ttvm3 - hudson.remoting.LocalChannel@146c0f Cloning the remote Git repository Cloning repository origin ERROR: Failed to clean the workspace java.io.IOException: Unable to delete /ttvm3 at hudson.Util.deleteFile(Util.java:237) at hudson.Util.deleteRecursive(Util.java:287) at hudson.FilePath$9.invoke(FilePath.java:856) at hudson.FilePath$9.invoke(FilePath.java:854) at hudson.FilePath.act(FilePath.java:788) at hudson.FilePath.act(FilePath.java:770) at hudson.FilePath.deleteRecursive(FilePath.java:854) at hudson.plugins.git.GitAPI.clone(GitAPI.java:205) at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1027) at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:968) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2099) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:287) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) ERROR: Error cloning remote repo 'origin' : Failed to delete workspace ERROR: Cause: Unable to delete /ttvm3 Trying next repository ERROR: Could not clone repository FATAL: Could not clone hudson.plugins.git.GitException: Could not clone at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1042) at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:968) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2099) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:287) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) {quote}
    via by tasat bar,
    • java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187 at hudson.Util.deleteFile(Util.java:254) at hudson.Util.deleteRecursive(Util.java:301) at hudson.Util.deleteContentsRecursive(Util.java:203) at hudson.Util.deleteRecursive(Util.java:300) at hudson.model.Run.delete(Run.java:1452) at hudson.tasks.LogRotator.perform(LogRotator.java:124) at hudson.model.Job.logRotate(Job.java:440) at hudson.model.Run.execute(Run.java:1739) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231)

    Users with the same issue

    Unknown visitor1 times, last one,