java.io.IOException: already closed

Jenkins JIRA | chad_lyon | 8 years ago
  1. 0

    We have a sort of special CI environment where after projects build we execute them remotely and use hudson to monitor their progress. The remote execution of these programs take a while and at certain points no output is sent back to the master for long periods of time. During these long intervals where no output is sent back (just over 2 hours) I am occasionally seeing the job fail with the following: FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:269) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:84) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:895) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request$1.get(Request.java:188) at hudson.remoting.Request$1.get(Request.java:157) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:261) ... 9 more Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request.abort(Request.java:223) at hudson.remoting.Channel.terminate(Channel.java:528) at hudson.remoting.Channel$ReaderThread.run(Channel.java:684) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:665) FATAL: Unable to delete script file /tmp/hudson24564.sh hudson.util.IOException2: remote file operation failed at hudson.FilePath.act(FilePath.java:544) at hudson.FilePath.delete(FilePath.java:741) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:895) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.FilePath.act(FilePath.java:541) ... 10 more FATAL: already closed java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:466) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:277) at hudson.model.Run.run(Run.java:895) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) However, this is not predictable or reproducible which makes me think it corresponds to an external event such as GC, or even an network or OS event (eg TCP Error or Socket timeout). Anyway I thought I would put it up here and see if anyone else is getting this too. I am using Hudson ver. 1.293, The master and slave are both RHEL 4 An interesting development occurred when I upgraded recently and then set hudson.util.ProcessTreeKiller.disable=true. The jobs were still failing but the underlying process was eventually completing its job successfully (copying a large MySQL DB if you must know). This is the reason I reported this. This hints at a bug in hudson's remoting code. --Chad

    Jenkins JIRA | 8 years ago | chad_lyon
    java.io.IOException: already closed
  2. 0

    We have a sort of special CI environment where after projects build we execute them remotely and use hudson to monitor their progress. The remote execution of these programs take a while and at certain points no output is sent back to the master for long periods of time. During these long intervals where no output is sent back (just over 2 hours) I am occasionally seeing the job fail with the following: FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:269) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:84) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:895) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request$1.get(Request.java:188) at hudson.remoting.Request$1.get(Request.java:157) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:261) ... 9 more Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request.abort(Request.java:223) at hudson.remoting.Channel.terminate(Channel.java:528) at hudson.remoting.Channel$ReaderThread.run(Channel.java:684) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:665) FATAL: Unable to delete script file /tmp/hudson24564.sh hudson.util.IOException2: remote file operation failed at hudson.FilePath.act(FilePath.java:544) at hudson.FilePath.delete(FilePath.java:741) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:895) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.FilePath.act(FilePath.java:541) ... 10 more FATAL: already closed java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:466) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:277) at hudson.model.Run.run(Run.java:895) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) However, this is not predictable or reproducible which makes me think it corresponds to an external event such as GC, or even an network or OS event (eg TCP Error or Socket timeout). Anyway I thought I would put it up here and see if anyone else is getting this too. I am using Hudson ver. 1.293, The master and slave are both RHEL 4 An interesting development occurred when I upgraded recently and then set hudson.util.ProcessTreeKiller.disable=true. The jobs were still failing but the underlying process was eventually completing its job successfully (copying a large MySQL DB if you must know). This is the reason I reported this. This hints at a bug in hudson's remoting code. --Chad

    Jenkins JIRA | 8 years ago | chad_lyon
    java.io.IOException: already closed
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    When there is an unexpected failure in the ssh connection, like: FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:269) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:84) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request$1.get(Request.java:188) at hudson.remoting.Request$1.get(Request.java:157) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:261) ... 9 more Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request.abort(Request.java:223) at hudson.remoting.Channel.terminate(Channel.java:528) at hudson.remoting.Channel$ReaderThread.run(Channel.java:697) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2498) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) at hudson.remoting.Channel$ReaderThread.run(Channel.java:678) FATAL: Unable to delete script file /tmp/hudson7182116958611275696.sh hudson.util.IOException2: remote file operation failed at hudson.FilePath.act(FilePath.java:596) at hudson.FilePath.delete(FilePath.java:793) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.FilePath.act(FilePath.java:593) ... 10 more FATAL: already closed java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:528) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:277) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) the processes on the slave keep on running.

    Jenkins JIRA | 8 years ago | Carlo de Wolf
    java.io.IOException: already closed
  5. 0

    When there is an unexpected failure in the ssh connection, like: FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:269) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:84) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request$1.get(Request.java:188) at hudson.remoting.Request$1.get(Request.java:157) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:261) ... 9 more Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request.abort(Request.java:223) at hudson.remoting.Channel.terminate(Channel.java:528) at hudson.remoting.Channel$ReaderThread.run(Channel.java:697) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2498) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) at hudson.remoting.Channel$ReaderThread.run(Channel.java:678) FATAL: Unable to delete script file /tmp/hudson7182116958611275696.sh hudson.util.IOException2: remote file operation failed at hudson.FilePath.act(FilePath.java:596) at hudson.FilePath.delete(FilePath.java:793) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.model.Build$RunnerImpl.build(Build.java:195) at hudson.model.Build$RunnerImpl.doRun(Build.java:151) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) Caused by: java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.FilePath.act(FilePath.java:593) ... 10 more FATAL: already closed java.io.IOException: already closed at hudson.remoting.Channel.send(Channel.java:342) at hudson.remoting.Request.call(Request.java:104) at hudson.remoting.Channel.call(Channel.java:481) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:528) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:277) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119) the processes on the slave keep on running.

    Jenkins JIRA | 8 years ago | Carlo de Wolf
    java.io.IOException: already closed

    7 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      already closed

      at hudson.remoting.Channel.send()
    2. Hudson :: Remoting Layer
      Channel.call
      1. hudson.remoting.Channel.send(Channel.java:342)
      2. hudson.remoting.Request.call(Request.java:104)
      3. hudson.remoting.Channel.call(Channel.java:481)
      3 frames
    3. Hudson
      Executor.run
      1. hudson.Launcher$RemoteLauncher.kill(Launcher.java:466)
      2. hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:277)
      3. hudson.model.Run.run(Run.java:895)
      4. hudson.model.Build.run(Build.java:112)
      5. hudson.model.ResourceController.execute(ResourceController.java:93)
      6. hudson.model.Executor.run(Executor.java:119)
      6 frames