java.net.ConnectException: Connection refused

Jenkins JIRA | Richard Mortimer | 2 years ago
  1. 0

    I have noticed that whenever I restart my Jenkins master my jnlp slaves are not reconnecting and require a manual slave restart to bring them back online. I've traced this back to the changes to fix JENKINS-19055. Specifically those changes cause the slave JVM to be restarted when the master disconnects. Prior to this change the remoting engine would wait for the server to restart before attempting to reconnect to the master. With the change it immediately tries to connect to the master and get a connection error because the master is restarting. This causes the slave to immediately terminate. Jenkins 1.575 gives the following slave log output when restarting the master {noformat} Aug 12, 2014 3:55:15 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Aug 12, 2014 3:55:15 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onDisconnect INFO: Restarting slave via jenkins.slaves.restarter.UnixSlaveRestarter@32a9f661 Aug 12, 2014 3:55:17 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: bishop Aug 12, 2014 3:55:17 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Aug 12, 2014 3:55:17 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins.example/] Aug 12, 2014 3:55:18 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: http://jenkins.example/tcpSlaveAgentListener/ is invalid: 503 Service Temporarily Unavailable java.lang.Exception: http://jenkins.example/tcpSlaveAgentListener/ is invalid: 503 Service Temporarily Unavailable at hudson.remoting.Engine.run(Engine.java:213) {noformat} Notice the "jenkins.slaves.restarter.JnlpSlaveRestarterInstaller" onDisconnect log message that performs a slave restart. Prior to JENKINS-19055 being integrated the slave called waitForServerToBack() repeatedly until the master came back online. For example {noformat} 25-Mar-2014 10:52:16 hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated 25-Mar-2014 10:52:26 hudson.remoting.Engine waitForServerToBack INFO: Failed to connect to the master. Will retry again java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385) at java.net.Socket.connect(Socket.java:546) at sun.net.NetworkClient.doConnect(NetworkClient.java:173) at sun.net.www.http.HttpClient.openServer(HttpClient.java:409) at sun.net.www.http.HttpClient.openServer(HttpClient.java:530) at sun.net.www.http.HttpClient.<init>(HttpClient.java:240) at sun.net.www.http.HttpClient.New(HttpClient.java:321) at sun.net.www.http.HttpClient.New(HttpClient.java:338) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801) at hudson.remoting.Engine.waitForServerToBack(Engine.java:371) at hudson.remoting.Engine.run(Engine.java:278) ... 25-Mar-2014 10:54:11 hudson.remoting.Engine waitForServerToBack INFO: Master isn't ready to talk to us. Will retry again: response code=503 25-Mar-2014 10:54:21 hudson.remoting.Engine waitForServerToBack INFO: Master isn't ready to talk to us. Will retry again: response code=503 25-Mar-2014 10:54:31 hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins.example/] 25-Mar-2014 10:54:31 hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins.example:42715 25-Mar-2014 10:54:31 hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking 25-Mar-2014 10:54:32 hudson.remoting.jnlp.Main$CuiListener status INFO: Connected {noformat} The connection/retry logic is contained in remoting Engine.java https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/Engine.java When connecting to the slave an error causes the connection to terminate (around line 232) {code} if(firstError!=null) { events.error(firstError); return; } {code} prior to JENKINS-19055 hooking into onDisconnect() a re-connection would not be attempted until waitForServerToBack() had ensured that the master had recovered. {code} events.onDisconnect(); // try to connect back to the server every 10 secs. waitForServerToBack(); {code} A quick and dirty fix would likely be to swap the onDisconnect and waitForServerToBack calls around.

    Jenkins JIRA | 2 years ago | Richard Mortimer
    java.net.ConnectException: Connection refused
  2. 0

    I have noticed that whenever I restart my Jenkins master my jnlp slaves are not reconnecting and require a manual slave restart to bring them back online. I've traced this back to the changes to fix JENKINS-19055. Specifically those changes cause the slave JVM to be restarted when the master disconnects. Prior to this change the remoting engine would wait for the server to restart before attempting to reconnect to the master. With the change it immediately tries to connect to the master and get a connection error because the master is restarting. This causes the slave to immediately terminate. Jenkins 1.575 gives the following slave log output when restarting the master {noformat} Aug 12, 2014 3:55:15 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Aug 12, 2014 3:55:15 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onDisconnect INFO: Restarting slave via jenkins.slaves.restarter.UnixSlaveRestarter@32a9f661 Aug 12, 2014 3:55:17 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: bishop Aug 12, 2014 3:55:17 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Aug 12, 2014 3:55:17 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins.example/] Aug 12, 2014 3:55:18 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: http://jenkins.example/tcpSlaveAgentListener/ is invalid: 503 Service Temporarily Unavailable java.lang.Exception: http://jenkins.example/tcpSlaveAgentListener/ is invalid: 503 Service Temporarily Unavailable at hudson.remoting.Engine.run(Engine.java:213) {noformat} Notice the "jenkins.slaves.restarter.JnlpSlaveRestarterInstaller" onDisconnect log message that performs a slave restart. Prior to JENKINS-19055 being integrated the slave called waitForServerToBack() repeatedly until the master came back online. For example {noformat} 25-Mar-2014 10:52:16 hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated 25-Mar-2014 10:52:26 hudson.remoting.Engine waitForServerToBack INFO: Failed to connect to the master. Will retry again java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385) at java.net.Socket.connect(Socket.java:546) at sun.net.NetworkClient.doConnect(NetworkClient.java:173) at sun.net.www.http.HttpClient.openServer(HttpClient.java:409) at sun.net.www.http.HttpClient.openServer(HttpClient.java:530) at sun.net.www.http.HttpClient.<init>(HttpClient.java:240) at sun.net.www.http.HttpClient.New(HttpClient.java:321) at sun.net.www.http.HttpClient.New(HttpClient.java:338) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801) at hudson.remoting.Engine.waitForServerToBack(Engine.java:371) at hudson.remoting.Engine.run(Engine.java:278) ... 25-Mar-2014 10:54:11 hudson.remoting.Engine waitForServerToBack INFO: Master isn't ready to talk to us. Will retry again: response code=503 25-Mar-2014 10:54:21 hudson.remoting.Engine waitForServerToBack INFO: Master isn't ready to talk to us. Will retry again: response code=503 25-Mar-2014 10:54:31 hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins.example/] 25-Mar-2014 10:54:31 hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins.example:42715 25-Mar-2014 10:54:31 hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking 25-Mar-2014 10:54:32 hudson.remoting.jnlp.Main$CuiListener status INFO: Connected {noformat} The connection/retry logic is contained in remoting Engine.java https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/Engine.java When connecting to the slave an error causes the connection to terminate (around line 232) {code} if(firstError!=null) { events.error(firstError); return; } {code} prior to JENKINS-19055 hooking into onDisconnect() a re-connection would not be attempted until waitForServerToBack() had ensured that the master had recovered. {code} events.onDisconnect(); // try to connect back to the server every 10 secs. waitForServerToBack(); {code} A quick and dirty fix would likely be to swap the onDisconnect and waitForServerToBack calls around.

    Jenkins JIRA | 2 years ago | Richard Mortimer
    java.net.ConnectException: Connection refused
  3. 0

    Activator fails to start when port 8888 is already bound.

    GitHub | 3 years ago | senia-psm
    java.net.ConnectException: Connection refused
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [elasticsearch 1.1] Marvel preventing cluster restart.

    GitHub | 3 years ago | hjz
    java.net.ConnectException: Connection refused
  6. 0

    Delay startup of marvel plugin

    GitHub | 3 years ago | markwalkom
    java.net.ConnectException: Connection refused

  1. tvrmsmith 1 times, last 3 weeks ago
  2. musketyr 2 times, last 2 months ago
  3. Malcshour 1 times, last 6 months ago
25 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.net.ConnectException

    Connection refused

    at java.net.PlainSocketImpl.socketConnect()
  2. Java RT
    HttpURLConnection.connect
    1. java.net.PlainSocketImpl.socketConnect(Native Method)
    2. java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
    3. java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
    4. java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
    5. java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
    6. java.net.Socket.connect(Socket.java:546)
    7. sun.net.NetworkClient.doConnect(NetworkClient.java:173)
    8. sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
    9. sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
    10. sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
    11. sun.net.www.http.HttpClient.New(HttpClient.java:321)
    12. sun.net.www.http.HttpClient.New(HttpClient.java:338)
    13. sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935)
    14. sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876)
    15. sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801)
    15 frames
  3. Hudson :: Remoting Layer
    Engine.run
    1. hudson.remoting.Engine.waitForServerToBack(Engine.java:371)
    2. hudson.remoting.Engine.run(Engine.java:278)
    2 frames