hudson.remoting.ChannelClosedException: channel is already closed

Jenkins JIRA | Johannes Wienke | 2 years ago
  1. 0

    Since approximately 9 months we have constant troubles with our master losing connectivity to our mac mavericks slave (via ssh). The issue we observe is that after some time, the master cannot communicate with the slave anymore, so that jobs fail with the following error message: {noformat} Building remotely on MAC_OS_mavericks_64bit (macos mavericks java7)FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:541) at hudson.remoting.Request.call(Request.java:129) at hudson.remoting.Channel.call(Channel.java:739) at hudson.EnvVars.getRemote(EnvVars.java:404) at hudson.model.Computer.getEnvironment(Computer.java:912) at jenkins.model.CoreEnvironmentContributor.buildEnvironmentFor(CoreEnvironmentContributor.java:29) at hudson.model.Run.getEnvironment(Run.java:2221) at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:885) at hudson.matrix.MatrixRun$MatrixRunExecution.decideWorkspace(MatrixRun.java:175) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1706) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:1027) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Caused by: java.util.concurrent.TimeoutException: Ping started on 1412069451954 hasn't completed at 1412069691955 ... 2 more {noformat} At some point, the slave is then marked as offline. When trying to reconnect, nothing happens. You see an empty log window with just the circling loading animation. No output is generated ever. We could not observer any issues with the underlying network connection. Everytime I observe this error, ssh-ing to the slave as the jenkins user is possible without any problems. This also only happens for the mavericks slave. All other Linux and Windows slave work perfectly. What is extremely confusing is that in case jenkins ended up in this condition, you cannot restart it in a clean fashion. You first have to kill the java process with SIGKILL, even though it is apparently not completely stuck since operation for everything apart from the mavericks slave continues to work perfectly. The general log file for jenkins only shows that also the jobs for checking disk space etc. suffer from the connectivity issue: {noformat} WARNING: Failed to monitor MAC_OS_mavericks_64bit for Architecture hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:541) at hudson.remoting.Request.callAsync(Request.java:208) at hudson.remoting.Channel.callAsync(Channel.java:766) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMo nitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDe scriptor.java:280) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:1027) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Caused by: java.util.concurrent.TimeoutException: Ping started on 1412069451954 hasn't complet ed at 1412069691955 ... 2 more {noformat} Apart from this, no errors are visible for that slave. A thread dump from the situation where the master tries to reconnect to the salve but nothing happens is available here: http://pastebin.com/DxFU8j7C

    Jenkins JIRA | 2 years ago | Johannes Wienke
    hudson.remoting.ChannelClosedException: channel is already closed
  2. 0

    [JENKINS-24926] Connection to slave constantly breaks, java process can only be kill -9-ed - Jenkins JIRA

    jenkins-ci.org | 6 months ago
    hudson.remoting.ChannelClosedException: channel is already closed
  3. 0

    Since approximately 9 months we have constant troubles with our master losing connectivity to our mac mavericks slave (via ssh). The issue we observe is that after some time, the master cannot communicate with the slave anymore, so that jobs fail with the following error message: {noformat} Building remotely on MAC_OS_mavericks_64bit (macos mavericks java7)FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:541) at hudson.remoting.Request.call(Request.java:129) at hudson.remoting.Channel.call(Channel.java:739) at hudson.EnvVars.getRemote(EnvVars.java:404) at hudson.model.Computer.getEnvironment(Computer.java:912) at jenkins.model.CoreEnvironmentContributor.buildEnvironmentFor(CoreEnvironmentContributor.java:29) at hudson.model.Run.getEnvironment(Run.java:2221) at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:885) at hudson.matrix.MatrixRun$MatrixRunExecution.decideWorkspace(MatrixRun.java:175) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1706) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:1027) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Caused by: java.util.concurrent.TimeoutException: Ping started on 1412069451954 hasn't completed at 1412069691955 ... 2 more {noformat} At some point, the slave is then marked as offline. When trying to reconnect, nothing happens. You see an empty log window with just the circling loading animation. No output is generated ever. We could not observer any issues with the underlying network connection. Everytime I observe this error, ssh-ing to the slave as the jenkins user is possible without any problems. This also only happens for the mavericks slave. All other Linux and Windows slave work perfectly. What is extremely confusing is that in case jenkins ended up in this condition, you cannot restart it in a clean fashion. You first have to kill the java process with SIGKILL, even though it is apparently not completely stuck since operation for everything apart from the mavericks slave continues to work perfectly. The general log file for jenkins only shows that also the jobs for checking disk space etc. suffer from the connectivity issue: {noformat} WARNING: Failed to monitor MAC_OS_mavericks_64bit for Architecture hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:541) at hudson.remoting.Request.callAsync(Request.java:208) at hudson.remoting.Channel.callAsync(Channel.java:766) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMo nitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDe scriptor.java:280) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:1027) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Caused by: java.util.concurrent.TimeoutException: Ping started on 1412069451954 hasn't complet ed at 1412069691955 ... 2 more {noformat} Apart from this, no errors are visible for that slave. A thread dump from the situation where the master tries to reconnect to the salve but nothing happens is available here: http://pastebin.com/DxFU8j7C

    Jenkins JIRA | 2 years ago | Johannes Wienke
    hudson.remoting.ChannelClosedException: channel is already closed
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [JENKINS-6817] FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel - Jenkins JIRA

    jenkins-ci.org | 11 months ago
    hudson.util.IOException2: remote file operation failed: /home/hudson/ci/jenkins/slaves/robusta/workspace/dp_10.0_marker_nightly at hudson.remoting.Channel@789e3ac5:robusta
  6. 0

    Jenkins issues - [JIRA] (JENKINS-14332) Repeated channel/timeout errors from Jenkins slave

    nabble.com | 1 year ago
    org.jenkinsci.lib.envinject.EnvInjectException: hudson.remoting.ChannelClosedException: channel is already closed

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.concurrent.TimeoutException

      Ping started on 1412069451954 hasn't completed at 1412069691955

      at hudson.remoting.PingThread.ping()
    2. Hudson :: Remoting Layer
      PingThread.run
      1. hudson.remoting.PingThread.ping(PingThread.java:120)
      2. hudson.remoting.PingThread.run(PingThread.java:81)
      2 frames