java.net.ConnectException: Connection refused

If you like a tip written by other Samebug users, mark is as helpful! Marks help our algorithm provide you better solutions and also help other users.
tip

This was a bug at HBase 1.0.1, it has been patched, so an update to 1.0.2 or further should solve the problem. URL for the Jira thread: https://goo.gl/c5oa6f

You have a different solution? A short tip here would help you and many other users who saw this issue last week.

  • puppet server's core components all assume that when they talk to the out-side world, they'll be talking IPv4 only. All of our internal machines are IPv6 only, so they resolve things like updates.puppetlabs.com and rubygems.org either directly to IPv6, or indirectly via DNS64 and NAT64. The first symptom of something being off is that puppetserver takes a long time to start up, and we only see in the debug log why: {code} 2015-02-26 17:26:46,048 DEBUG [p.s.v.version-check-core] Could not retrieve update information (http://updates.puppetlabs.com) java.net.SocketException: Network is unreachable at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_31] at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_31] at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_31] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_31] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:272) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:141) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[puppet-server-release.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31] {code} the second can be seen when trying to install a gem: {code} igalic@puppet01 ~ % sudo -H /usr/bin/puppetserver gem install hiera-file ERROR: Could not find a valid gem 'hiera-file' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SocketError: Network is unreachable (https://rubygems.org/latest_specs.4.8.gz) sudo -H /usr/bin/puppetserver gem install hiera-file 13,43s user 0,31s system 182% cpu 7,511 total igalic@puppet01 ~ % {code} i'd like to be able to tell puppetserver to prefer ipv6. putting that intent in the form of {{JAVA_ARGS="-Xms2g -Xmx2g -Djava.net.preferIPv6Addresses=true"}} into {{/etc/default/puppetserver}} doesn't seem to suffice: startup: {code} 2015-02-26 17:34:56,866 DEBUG [p.s.v.version-check-core] Could not retrieve update information (http://updates.puppetlabs.com) java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_31] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_31] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38) ~[puppet-server-release.jar:na] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57) ~[puppet-server-release.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31] {code} gem: {code} igalic@puppet01 ~ % sudo -H /usr/bin/puppetserver gem install hiera-file ERROR: Could not find a valid gem 'hiera-file' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SocketError: Network is unreachable (https://rubygems.org/latest_specs.4.8.gz) sudo -H /usr/bin/puppetserver gem install hiera-file 13,97s user 0,33s system 187% cpu 7,637 total igalic@puppet01 ~ % {code}
    via by Igor Galić,
  • We have encountered a few issues with handling an HttpAsyncClient request which is made when the process has too many open file descriptors: 1) A .get() call made on the future returned from the httpclient's .execute() method hangs indefinitely. 2) No call is made to any of the methods on the {{FutureCallback<HttpResponse>}} object provided as an argument to the httpclient's .execute() method. I was expecting a call to .failed() to occur. 3) A call to the httpclient's .close() method does not error out but also does not result in all of the IOReactor "I/O Dispatcher" threads being shut down. In normal cases, these threads would get shut down when the .close() call is made. I've attached a couple of Java source files which demonstrate the problem - AbnormalIOReactor.java and ResponseCallback.java. AbnormalIOReactor.java is based on the example at https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchange.java. Just after the httpclient is started, this class allocates bare sockets repeatedly and holds onto them. This is done to force the application to get into the state where file descriptors have been exhausted. The httpclient.execute() call does not throw an error. The class repeatedly calls .get() on the future response with a prolonged timeout, breaking out of the loop when the client moves from the state of having .isRunning() be true to .isRunning() being false. Checking for a change in the .isRunning() state is a bit contrived here, ideally something that an API client would not have to do. I noticed from the implementation of the {{CloseableHttpAsyncClientBase}} constructor at https://svn.apache.org/repos/asf/httpcomponents/httpasyncclient/tags/4.1.1/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java that the reactor thread's run() method will catch an exception at request startup and set the client status to {{STOPPED}} and this seemed like the only way that a client could detect that a problem had occurred in this situation. {code:java} @Override public void run() { try { final IOEventDispatch ioEventDispatch = new InternalIODispatch(handler); connmgr.execute(ioEventDispatch); } catch (final Exception ex) { log.error("I/O reactor terminated abnormally", ex); } finally { status.set(Status.STOPPED); } } {code} The implementation of ResponseCallback.java writes a message to stdout when any of the methods is invoked. In this case, no message is written to stdout even though the request has effectively failed. When this program is run from the command line like {{java abnormalioreactor.AbnormalIOReactor}}, it will hang after the word "Done" appears, which occurs after the .close() method is called on the httpclient. When running jstack from the command line, I see that various "I/O Dispatcher" threads are still running for the client even though I had expected them to have been closed: {noformat} "I/O dispatcher 8" #19 prio=5 os_prio=31 tid=0x00007f978e800000 nid=0x6503 runnable [0x0000000120cf6000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000076ba79bf8> (a sun.nio.ch.Util$2) - locked <0x000000076ba79be8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000076ba79ac8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) at java.lang.Thread.run(Thread.java:745) {noformat} I had been thinking that in this situation it would be better for the httpclient to not be implicitly stopped - so it could be used for subsequent requests if/when the number of open file descriptors has dropped. Also, it would be good for either the httpclient.execute() call to throw an exception right away or at least for calls to .get() on the future returned from the .execute() call to throw an exception and for the .failed() method on {{FutureCallback<HttpResponse>}} object to be called. Note that I ran this application on my MacBook Pro, running OS-X Yosemite. We've seen this problem on other OSes as well, though, including CentOS 7. I reproduced this problem with a few different versions of HttpAsyncClient - 4.0.2, 4.1.1, custom build from trunk, custom build from the 4.1.x branch. This isn't a situation that we run into often with our applications - mostly in cases where users haven't tuned their application settings properly to account for the number of file descriptors that a Java process would need to use. It would be nice if the HttpAsyncClient handling could gracefully fail a request when open file descriptors have temporarily been exhausted under load spikes but leave the client in a usable state for future requests.
    via by Jeremy Barlow,
    • java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)[na:1.8.0_31] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)[na:1.8.0_31] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173)[puppet-server-release.jar:na] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147)[puppet-server-release.jar:na] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)[puppet-server-release.jar:na] at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)[puppet-server-release.jar:na] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)[puppet-server-release.jar:na] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)[puppet-server-release.jar:na] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)[puppet-server-release.jar:na] at java.lang.Thread.run(Thread.java:745)[na:1.8.0_31]

    Users with the same issue

    rocday
    rocday2 times, last one,
    Unknown visitor1 times, last one,
    Hronom
    Hronom2 times, last one,
    Alireza Mohamadi
    Alireza Mohamadi142 times, last one,
    Unknown visitor1 times, last one,
    114 more bugmates