java.util.concurrent.TimeoutException: Ping started on 1349293259718 hasn't completed at 1349293499718

Jenkins JIRA | Krzysztof Malinowski | 4 years ago
  1. 0

    I have set up Jenkins slave Windows service so that it is restarted by system on failure. It works OK if slave fails for whatever reasons. Unfortunately recently master keeps failing due to out of memory problems and after master reboot some Windows slave do not reconnect. Service has to be restarted in order to connect slave again. {panel:title=jenkins-slave.err.log} Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Hudson agent is running in headless mode. Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 02, 2012 3:59:11 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 03, 2012 8:11:21 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 03, 2012 8:15:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 03, 2012 9:44:59 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel. java.util.concurrent.TimeoutException: Ping started on 1349293259718 hasn't completed at 1349293499718 at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Oct 03, 2012 9:44:59 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.net.SocketException: socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:90) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Oct 03, 2012 9:44:59 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Hudson agent is running in headless mode. Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected {panel} {panel:title=jenkins-slave.wrapper.log} 2012-10-02 15:59:10 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://server:8080/computer/slave-1/slave-agent.jnlp 2012-10-02 15:59:10 - Started 5680 2012-10-04 16:20:05 - Stopping jenkinsslave-C__Jenkins 2012-10-04 16:20:05 - ProcessKill 5680 2012-10-04 16:20:05 - Finished jenkinsslave-C__Jenkins 2012-10-04 16:20:06 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://server:8080/computer/slave-1/slave-agent.jnlp 2012-10-04 16:20:06 - Started 6464 {panel} Note that before exception slave itself reconnected many times to master, but after exception it did not recover until service was restarted.

    Jenkins JIRA | 4 years ago | Krzysztof Malinowski
    java.util.concurrent.TimeoutException: Ping started on 1349293259718 hasn't completed at 1349293499718
  2. 0

    I have set up Jenkins slave Windows service so that it is restarted by system on failure. It works OK if slave fails for whatever reasons. Unfortunately recently master keeps failing due to out of memory problems and after master reboot some Windows slave do not reconnect. Service has to be restarted in order to connect slave again. {panel:title=jenkins-slave.err.log} Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Hudson agent is running in headless mode. Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 02, 2012 3:59:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 02, 2012 3:59:11 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 03, 2012 8:11:21 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 03, 2012 8:12:22 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 03, 2012 8:15:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 03, 2012 8:15:59 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 03, 2012 9:44:59 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel. java.util.concurrent.TimeoutException: Ping started on 1349293259718 hasn't completed at 1349293499718 at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Oct 03, 2012 9:44:59 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.net.SocketException: socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:90) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Oct 03, 2012 9:44:59 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Hudson agent is running in headless mode. Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://server:8080/] Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to server:21234 Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 04, 2012 4:20:07 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected {panel} {panel:title=jenkins-slave.wrapper.log} 2012-10-02 15:59:10 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://server:8080/computer/slave-1/slave-agent.jnlp 2012-10-02 15:59:10 - Started 5680 2012-10-04 16:20:05 - Stopping jenkinsslave-C__Jenkins 2012-10-04 16:20:05 - ProcessKill 5680 2012-10-04 16:20:05 - Finished jenkinsslave-C__Jenkins 2012-10-04 16:20:06 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://server:8080/computer/slave-1/slave-agent.jnlp 2012-10-04 16:20:06 - Started 6464 {panel} Note that before exception slave itself reconnected many times to master, but after exception it did not recover until service was restarted.

    Jenkins JIRA | 4 years ago | Krzysztof Malinowski
    java.util.concurrent.TimeoutException: Ping started on 1349293259718 hasn't completed at 1349293499718
  3. 0

    ERROR: Processing failed due to a bug in the code Jenkins 1.509.2

    Google Groups | 3 years ago | Mehdi Hayani Soujaa
    java.util.concurrent.TimeoutException: Ping started on 1378290933654 hasn't completed at 1378291173655
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Slave stops responding for whatever reason. First ping thread notices slave does not respond: Nov 26, 2014 2:00:59 PM INFO hudson.slaves.ChannelPinger$1 onDead Ping failed. Terminating the channel. java.util.concurrent.TimeoutException: Ping started on 1417006619758 hasn't completed at 1417006859758 at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) (would be nice if we could see which channel/slave) A bit later: Nov 26, 2014 2:14:34 PM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor SV-ARG-DEV-D23 for Clock Difference hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:541) at hudson.remoting.Request.callAsync(Request.java:208) at hudson.remoting.Channel.callAsync(Channel.java:766) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:280) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:1027) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) at hudson.remoting.PingThread.ping(PingThread.java:120) at hudson.remoting.PingThread.run(PingThread.java:81) Caused by: java.util.concurrent.TimeoutException: Ping started on 1417006619758 hasn't completed at 1417006859758 ... 2 more Yet node is not shown as offline, and still shows as executor busy. Threaddump shows executor thread on master: Executor #0 for DWI01164 : executing OKA.R201501/Team D/_publics/Public - 4 #103 / waiting for hudson.remoting.Channel@3b91e986:DWI01164 "Executor #0 for DWI01164 : executing OKA.R201501/Team D/_publics/Public - 4 #103 / waiting for hudson.remoting.Channel@3b91e986:DWI01164" Id=16260 Group=main TIMED_WAITING on hudson.remoting.UserRequest@e4ca0c4 at java.lang.Object.wait(Native Method) - waiting on hudson.remoting.UserRequest@e4ca0c4 at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:739) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168) at com.sun.proxy.$Proxy56.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956) at hudson.Launcher$ProcStarter.join(Launcher.java:367) at hudson.tasks.Maven.perform(Maven.java:328) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756) at hudson.model.Build$BuildExecution.build(Build.java:198) at hudson.model.Build$BuildExecution.doRun(Build.java:159) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529) at hudson.model.Run.execute(Run.java:1706) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) It looks like ChannelPinger calls channel.close() which sets channel.outClosed , yet Request is stuck in a loop checking for channel.isInClosed(), which was not set. Should Request check for channel.isClosingOrClosed()?

    Jenkins JIRA | 2 years ago | Wannes Sels
    java.util.concurrent.TimeoutException: Ping started on 1417006619758 hasn't completed at 1417006859758
  6. 0

    [JENKINS-24926] Connection to slave constantly breaks, java process can only be kill -9-ed - Jenkins JIRA

    jenkins-ci.org | 5 months ago
    java.util.concurrent.TimeoutException: Ping started on 1413132699262 hasn't completed at 1413132939263

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.concurrent.TimeoutException

      Ping started on 1349293259718 hasn't completed at 1349293499718

      at hudson.remoting.PingThread.ping()
    2. Hudson :: Remoting Layer
      PingThread.run
      1. hudson.remoting.PingThread.ping(PingThread.java:120)
      2. hudson.remoting.PingThread.run(PingThread.java:81)
      2 frames