java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

Apache's JIRA Issue Tracker | Christopher Tubbs | 9 months ago
  1. 0

    [jira] [Resolved] (HADOOP-12918) MiniDFSCluster uses wrong IP address

    hadoop-common-dev | 9 months ago | Christopher Tubbs (JIRA)
    java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
  2. 0

    MiniDFSCluster seems to be registering the DataNode using the machine's internal IP address, rather than "localhost/127.0.0.1". It looks like the problem isn't MiniDFSCluster specific, but that's what's biting me right now and I can't figure out a workaround. MiniDFSCluster logs show roughly the following (jetty services ignored): NameNode starts org.apache.hadoop.ipc.Server listening on localhost/127.0.0.1:43023 DataNode reports "Configured hostname is 127.0.0.1" DataNode reports "Opened streaming server at /127.0.0.1:57310" DataNode starts org.apache.hadoop.ipc.Server listening on localhost/127.0.0.1:53015 DataNode registers with NN using storage id DS-XXXXXXXXX-172.31.3.214-57310-XXXXXXXXXXXXX with ipcPort=53015 NameNode reports "Adding a new node: /default-rack/172.31.3.214:57310" The storage id should have been derived from 127.0.0.1, and the so should all the other registered information. I've verified with netstat that all services were listening only on 127.0.0.1 This resulted in the client being unable to write blocks to the datanode, because it was not listening on the address given to it by the namenode (the address it was registered under). The actual client error message is: {code:java} [IPC Server handler 0 on 43023} INFO org.apache.hadoop.hdfs.StateChange - BLOCK* allocateBlock: /test-dir/HelloWorld.jar. BP-460569874-172.31.3.214-1457727894640 blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.31.3.214:57310|RBW]]} [Thread-61} INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1128) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1088) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514) [Thread-61} INFO org.apache.hadoop.hdfs.DFSClient - Abandoning BP-460569874-172.31.3.214-1457727894640:blk_1073741825_1001 [Thread-61} INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 172.31.3.214:57310 [IPC Server handler 2 on 43023} WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy - Not able to place enough replicas, still in need of 1 to reach 1 For more information, please enable DEBUG log level on org.apache.commons.logging.impl.Log4JLogger [IPC Server handler 2 on 43023} ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:christopher (auth:SIMPLE) cause:java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. [IPC Server handler 2 on 43023} INFO org.apache.hadoop.ipc.Server - IPC Server handler 2 on 43023, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.3.214:57395 Call#12 Retry#0: error: java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) [Thread-61} WARN org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy17.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy17.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514) {code} Additional information: I've tried with Hadoop 2.2.0, 2.6.1, and 2.6.3 and same results. It probably affects other versions. I do not see this problem running locally, only in EC2, but I've yet to be able to find a relevant networking configuration difference which would have any effect. (no extra entries in /etc/hosts, no DNS issues, etc.) I can reproduce this easily by trying to build Accumulo's master branch (HEAD at db21315) with `mvn clean package -Dtest=VfsClassLoaderTest -DfailIfNoTests=false -Dhadoop.version=2.6.3`

    Apache's JIRA Issue Tracker | 9 months ago | Christopher Tubbs
    java.io.IOException: File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
  3. 0

    HDFS Under replicated blocks

    Stack Overflow | 3 years ago | SuvP
    java.io.IOException: file /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2013_10_21-15_33_53 on client 111.222.333.444 Requested replication 3 exceeds maximum 1
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    EMR Cluster terminates when trying to start a cluster with 2 nodes

    Stack Overflow | 2 years ago | Pablo Natal
    java.io.IOException: File /mnt/var/em/verify-hdfs-access-1147553264033 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
  6. 0

    SSH Tunnel to access EC2 Hadoop Cluster

    Stack Overflow | 3 years ago | Dhams
    java.io.IOException: File /user/ubuntu/features.json could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      File /test-dir/HelloWorld.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget()
    2. Apache Hadoop HDFS
      ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod
      1. org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
      2. org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
      3. org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
      4. org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
      5. org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
      5 frames
    3. Hadoop
      Server$Handler$1.run
      1. org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
      2. org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
      3. org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
      4. org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
      4 frames
    4. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:422)
      2 frames
    5. Hadoop
      Server$Handler.run
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
      2. org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
      2 frames