java.util.concurrent.TimeoutException: Ping started at 1442902614156 hasn't completed by 1442902854206

Jenkins JIRA | Hans Baer | 1 year ago
  1. 0

    Almost on daily basis my Jenkins is shutting down is taking ALL slaves offline. The reasons for this is unknown to me and looks like a severe bug. Can you please help to check this? Based on my observation I notice that connecting new slaves seems to fail with an SSL exception. ---- Sep 22, 2015 8:08:42 AM org.eclipse.jetty.util.log.JavaUtilLog warn WARNING: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source) at sun.nio.ch.SocketChannelImpl.write(Unknown Source) at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:293) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:402) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:337) at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:48) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:738) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.shutdownOutput(SslConnection.java:641) at org.eclipse.jetty.io.nio.SslConnection.onIdleExpired(SslConnection.java:260) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.onIdleExpired(SelectChannelEndPoint.java:349) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:326) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Sep 22, 2015 8:08:48 AM org.eclipse.jetty.util.log.JavaUtilLog warn WARNING: handle failed java.lang.IllegalStateException: Internal error at sun.security.ssl.SSLEngineImpl.initHandshaker(Unknown Source) at sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source) at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source) at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source) at javax.net.ssl.SSLEngine.unwrap(Unknown Source) at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:536) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:401) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:193) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ---- Shortly afterwards I can see that Jenkins is taking ALL slaves offline ---- Sep 22, 2015 8:20:54 AM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel SLAVE-101051. java.util.concurrent.TimeoutException: Ping started at 1442902614156 hasn't completed by 1442902854206 at hudson.remoting.PingThread.ping(PingThread.java:126) at hudson.remoting.PingThread.run(PingThread.java:85) ---- Afterwards ALL slaves want to register back to Jenkins but Jenkins is rejecting it with ---- INFO: Accepted connection #288 from /10.0.209.109:64213 Sep 22, 2015 8:47:00 AM jenkins.slaves.JnlpSlaveHandshake error WARNING: TCP slave agent connection handler #288 with /10.0.209.109:64213 is aborted: SLAVE-719161 is already connected to this master. Rejecting this connection. Sep 22, 2015 8:47:00 AM hudson.TcpSlaveAgentListener$ConnectionHandler run ---- If Jenkins kicks out all slaves, I would expect Jenkins to allow it get automatically accepted again instead of referring to already existing connection. But that all slaves are being taken offline at once due to PING FAIL is rather a bug. Please find full logs attached as well!

    Jenkins JIRA | 1 year ago | Hans Baer
    java.util.concurrent.TimeoutException: Ping started at 1442902614156 hasn't completed by 1442902854206
  2. 0

    Almost on daily basis my Jenkins is shutting down is taking ALL slaves offline. The reasons for this is unknown to me and looks like a severe bug. Can you please help to check this? Based on my observation I notice that connecting new slaves seems to fail with an SSL exception. ---- Sep 22, 2015 8:08:42 AM org.eclipse.jetty.util.log.JavaUtilLog warn WARNING: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source) at sun.nio.ch.SocketChannelImpl.write(Unknown Source) at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:293) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:402) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:337) at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:48) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:738) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.shutdownOutput(SslConnection.java:641) at org.eclipse.jetty.io.nio.SslConnection.onIdleExpired(SslConnection.java:260) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.onIdleExpired(SelectChannelEndPoint.java:349) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:326) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Sep 22, 2015 8:08:48 AM org.eclipse.jetty.util.log.JavaUtilLog warn WARNING: handle failed java.lang.IllegalStateException: Internal error at sun.security.ssl.SSLEngineImpl.initHandshaker(Unknown Source) at sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source) at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source) at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source) at javax.net.ssl.SSLEngine.unwrap(Unknown Source) at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:536) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:401) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:193) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ---- Shortly afterwards I can see that Jenkins is taking ALL slaves offline ---- Sep 22, 2015 8:20:54 AM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel SLAVE-101051. java.util.concurrent.TimeoutException: Ping started at 1442902614156 hasn't completed by 1442902854206 at hudson.remoting.PingThread.ping(PingThread.java:126) at hudson.remoting.PingThread.run(PingThread.java:85) ---- Afterwards ALL slaves want to register back to Jenkins but Jenkins is rejecting it with ---- INFO: Accepted connection #288 from /10.0.209.109:64213 Sep 22, 2015 8:47:00 AM jenkins.slaves.JnlpSlaveHandshake error WARNING: TCP slave agent connection handler #288 with /10.0.209.109:64213 is aborted: SLAVE-719161 is already connected to this master. Rejecting this connection. Sep 22, 2015 8:47:00 AM hudson.TcpSlaveAgentListener$ConnectionHandler run ---- If Jenkins kicks out all slaves, I would expect Jenkins to allow it get automatically accepted again instead of referring to already existing connection. But that all slaves are being taken offline at once due to PING FAIL is rather a bug. Please find full logs attached as well!

    Jenkins JIRA | 1 year ago | Hans Baer
    java.util.concurrent.TimeoutException: Ping started at 1442902614156 hasn't completed by 1442902854206
  3. 0

    My only Jenkins slave dies every day almost at the same time giving below error : {code:java} Jul 29, 2016 3:02:42 AM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1469775521795 hasn't completed by 1469775762018 at hudson.remoting.PingThread.ping(PingThread.java:126) at hudson.remoting.PingThread.run(PingThread.java:85) Jul 29, 2016 3:02:46 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Jul 29, 2016 3:02:47 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Jul 29, 2016 3:03:00 AM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@e374354 Jul 29, 2016 3:03:39 AM hudson.remoting.jnlp.Main createEngine INFO: Trying protocol: JNLP2-connect Jul 29, 2016 3:03:41 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: ICI-Internal_1 is already connected to this master. Rejecting this connection. {code} AT the same time, Jenkins Server (master) exhibits following logs: {code:java} Jul 29, 2016 3:02:37 AM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel ICI-Internal_1. java.util.concurrent.TimeoutException: Ping started at 1469775517149 hasn't completed by 1469775757149 at hudson.remoting.PingThread.ping(PingThread.java:126) at hudson.remoting.PingThread.run(PingThread.java:85) Jul 29, 2016 3:03:40 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #2 from /12.170.11.58:46412 Jul 29, 2016 3:03:41 AM org.jenkinsci.remoting.engine.JnlpServerHandshake error WARNING: TCP agent connection handler #2 with /12.170.11.58:46412 is aborted: ICI-Internal_1 is already connected to this master. Rejecting this connection. Jul 29, 2016 3:03:41 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #3 from /12.170.11.58:46413 Jul 29, 2016 3:03:41 AM org.jenkinsci.remoting.engine.JnlpServerHandshake error WARNING: TCP agent connection handler #3 with /12.170.11.58:46413 is aborted: ICI-Internal_1 is already connected to this master. Rejecting this connection. {code} I only understood that Client/Server both are not able to connect to each other due to some reason and failed. However, it works for rest of the day when it is started manually after above error. There is no firewall/selinux running on any of the server. -Amit

    Jenkins JIRA | 4 months ago | Amit Naudiyal
    java.util.concurrent.TimeoutException: Ping started at 1469775521795 hasn't completed by 1469775762018
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [JIRA] [core] (JENKINS-30587) All slaves get terminated without reconnection possibility.

    Google Groups | 1 year ago | worldc...@yahoo.de (JIRA)
    java.util.concurrent.TimeoutException: Ping started at 1442902614156 hasn't completed by 1442902854206<br />
  6. 0

    [JIRA] [monitoring-plugin] (JENKINS-20947) Failed to monitor for Free Swap Space

    Google Groups | 1 year ago | stephen...@java.net (JIRA)
    java.util.concurrent.TimeoutException: Ping started at 1462866895203 hasn't completed by 1462867135208<br />

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.concurrent.TimeoutException

      Ping started at 1442902614156 hasn't completed by 1442902854206

      at hudson.remoting.PingThread.ping()
    2. Hudson :: Remoting Layer
      PingThread.run
      1. hudson.remoting.PingThread.ping(PingThread.java:126)
      2. hudson.remoting.PingThread.run(PingThread.java:85)
      2 frames