java.net.ConnectException: Connection refused

Apache's JIRA Issue Tracker | Christopher Tubbs | 9 months ago
  1. 0

    [jira] [Resolved] (HADOOP-12918) MiniDFSCluster uses wrong IP address

    hadoop-common-dev | 9 months ago | Christopher Tubbs (JIRA)
    java.net.ConnectException: Connection refused
  2. 0

    MiniDFSCluster seems to be registering the DataNode using the machine's internal IP address, rather than "localhost/127.0.0.1". It looks like the problem isn't MiniDFSCluster specific, but that's what's biting me right now and I can't figure out a workaround. MiniDFSCluster logs show roughly the following (jetty services ignored): NameNode starts org.apache.hadoop.ipc.Server listening on localhost/127.0.0.1:43023 DataNode reports "Configured hostname is 127.0.0.1" DataNode reports "Opened streaming server at /127.0.0.1:57310" DataNode starts org.apache.hadoop.ipc.Server listening on localhost/127.0.0.1:53015 DataNode registers with NN using storage id DS-XXXXXXXXX-172.31.3.214-57310-XXXXXXXXXXXXX with ipcPort=53015 NameNode reports "Adding a new node: /default-rack/172.31.3.214:57310" The storage id should have been derived from 127.0.0.1, and the so should all the other registered information. I've verified with netstat that all services were listening only on 127.0.0.1 This resulted in the client being unable to write blocks to the datanode, because it was not listening on the address given to it by the namenode (the address it was registered under). The actual client error message is: {code:java} [IPC Server handler 0 on 43023} INFO org.apache.hadoop.hdfs.StateChange - BLOCK* allocateBlock: /test-dir/HelloWorld.jar. BP-460569874-172.31.3.214-1457727894640 blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.31.3.214:57310|RBW]]} [Thread-61} INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1128) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514) [Thread-61} INFO org.apache.hadoop.hdfs.DFSClient - Abandoning BP-460569874-172.31.3.214-1457727894640:blk_1073741825_1001 [Thread-61} INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 172.31.3.214:57310 [IPC Server handler 2 on 43023} WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy - Not able to place enough replicas, still in need of 1 to reach 1 For more information, please enable DEBUG log level on org.apache.commons.logging.impl.Log4JLogger [IPC Server handler 2 on 43023} ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:christopher (auth:SIMPLE) cause:java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. [IPC Server handler 2 on 43023} INFO org.apache.hadoop.ipc.Server - IPC Server handler 2 on 43023, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.3.214:57395 Call#12 Retry#0: error: java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) [Thread-61} WARN org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy17.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy17.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514) {code} Additional information: I've tried with Hadoop 2.2.0, 2.6.1, and 2.6.3 and same results. It probably affects other versions. I do not see this problem running locally, only in EC2, but I've yet to be able to find a relevant networking configuration difference which would have any effect. (no extra entries in /etc/hosts, no DNS issues, etc.) I can reproduce this easily by trying to build Accumulo's master branch (HEAD at db21315) with `mvn clean package -Dtest=VfsClassLoaderTest -DfailIfNoTests=false -Dhadoop.version=2.6.3`

    Apache's JIRA Issue Tracker | 9 months ago | Christopher Tubbs
    java.net.ConnectException: Connection refused
  3. 0

    Spark 1.2 cannot connect to HDFS on HDP 2.2

    Stack Overflow | 2 years ago | John
    java.net.ConnectException: Connection refused
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Executing 'create table' statement always hangs, It looks like a previous problem unsolved.

    Google Groups | 3 years ago | David
    java.net.ConnectException: Connection refused
  6. 0

    hdfs dfs -copyFromLocal Datanode refusing connection

    Stack Overflow | 9 months ago | Hitesh Agrawal
    java.net.ConnectException: Connection refused

  1. RoiOtero 525 times, last 2 weeks ago
  2. nasimk 1595 times, last 3 days ago
  3. gehel 8 times, last 1 month ago
  4. rp 1 times, last 2 months ago
  5. poroszd 4 times, last 2 months ago
10 more registered users
84 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.net.ConnectException

    Connection refused

    at sun.nio.ch.SocketChannelImpl.checkConnect()
  2. Java RT
    SocketChannelImpl.finishConnect
    1. sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    2. sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    2 frames
  3. Hadoop
    NetUtils.connect
    1. org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    2. org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
    2 frames
  4. Apache Hadoop HDFS
    DFSOutputStream$DataStreamer.run
    1. org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305)
    2. org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1128)
    3. org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088)
    4. org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
    4 frames