java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526]

Spring JIRA | Thomas Risberg | 2 years ago
  1. 0

    Configure hdfs-site.xml with the following properties: {code} <property> <name>dfs.socket.timeout</name> <value>20000</value> </property> <property> <name>dfs.datanode.socket.write.timeout</name> <value>20000</value> </property> {code} then write to a file with a timeout higher than 20000, and let it time out, should see something like this on the datanode logs: {code} 2014-10-13 14:49:33,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1412236653-127.0.0.1-1404394045731:blk_1073751606_10782, type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted. 2014-10-13 14:49:33,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1412236653-127.0.0.1-1404394045731:blk_1073751606_10782, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2014-10-13 14:49:33,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-1412236653-127.0.0.1-1404394045731:blk_1073751606_10782 received exception java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526] 2014-10-13 14:49:33,324 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: carbon:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.0.110:56526 dest: /192.168.0.111:50010 java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221) at java.lang.Thread.run(Thread.java:701) {code} On the client side we should see something like this: {code} java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1492) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:116) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:721) 14:49:59,813 ERROR taskExecutor-1 output.TextFileWriter - error closing java.io.IOException: All datanodes 192.168.0.111:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1008) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475) {code}

    Spring JIRA | 2 years ago | Thomas Risberg
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526]
  2. 0

    Configure hdfs-site.xml with the following properties: {code} <property> <name>dfs.socket.timeout</name> <value>20000</value> </property> <property> <name>dfs.datanode.socket.write.timeout</name> <value>20000</value> </property> {code} then write to a file with a timeout higher than 20000, and let it time out, should see something like this on the datanode logs: {code} 2014-10-13 14:49:33,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1412236653-127.0.0.1-1404394045731:blk_1073751606_10782, type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted. 2014-10-13 14:49:33,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1412236653-127.0.0.1-1404394045731:blk_1073751606_10782, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2014-10-13 14:49:33,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-1412236653-127.0.0.1-1404394045731:blk_1073751606_10782 received exception java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526] 2014-10-13 14:49:33,324 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: carbon:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.0.110:56526 dest: /192.168.0.111:50010 java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221) at java.lang.Thread.run(Thread.java:701) {code} On the client side we should see something like this: {code} java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1492) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:116) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:721) 14:49:59,813 ERROR taskExecutor-1 output.TextFileWriter - error closing java.io.IOException: All datanodes 192.168.0.111:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1008) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475) {code}

    Spring JIRA | 2 years ago | Thomas Risberg
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526]
  3. 0

    Problem syncing commit log: /hypertable/servers/rs18/log/user/3252: Error flushing DFS fd 73135

    Google Groups | 3 years ago | David
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.190.115.66:50010 remote=/10.190.115.56:52631]
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    RegionServers Crashing every hour in production env

    Google Groups | 4 years ago | Pablo Musa
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.17.2.18:50010 remote=/172.17.2.18:37074]
  6. 0

    reduce端缓存数据过多出现FGC,导致reduce生成的数据无法写到hdfs - 快乐程序员的专栏 - 博客频道 - CSDN.NET

    csdn.net | 1 year ago
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.39.5.193:50010 remote=/10.39.5.185:58225]

    2 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.net.SocketTimeoutException

      20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.111:50010 remote=/192.168.0.110:56526]

      at org.apache.hadoop.net.SocketIOWithTimeout.doIO()
    2. Hadoop
      SocketInputStream.read
      1. org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
      2. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
      3. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
      3 frames
    3. Java RT
      DataInputStream.read
      1. java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
      2. java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
      3. java.io.BufferedInputStream.read(BufferedInputStream.java:334)
      4. java.io.DataInputStream.read(DataInputStream.java:149)
      4 frames
    4. Hadoop
      IOUtils.readFully
      1. org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
      1 frame
    5. Apache Hadoop HDFS
      DataXceiver.run
      1. org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
      2. org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
      3. org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
      4. org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435)
      5. org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693)
      6. org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569)
      7. org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
      8. org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
      9. org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
      9 frames
    6. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:701)
      1 frame