java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.109.61:50010 remote=/192.168.109.11:46349]

Spring JIRA | Sina Sojoodi | 2 years ago
  1. 0

    Environment: - Hadoop Installation: PHD Service for PCF (PHD1.1 based on Apache Hadoop 2.0.5: 2.0.5-alpha-gphd-2.1.0.0 ) running on vCHS - Spring XD running in singlenode mode (version 1.0.0.RC1) on a vCHS VM Steps to reproduce: 1- Setup a stream in Spring XD shell: "http --port=9000 | hdfs --rollover=10M --idleTimeout=60000" --deploy 2- Hit port 9000 every second with 1-10KB of JSON data 3- Observe the temp file being created in HDFS under /xd/<stream name> 4- Run `hadoop fs tail <file> --follow` to see that data is being written to HDFS Expected result: - HDFS sink continues to operate and eventually roll-over at 10MB Actual: - After about 2 minutes of successful HDFS writes, the HDFS sink crashes and starts throwing exceptions (see full log attached): "'java.io.IOException: All datanodes 192.168.109.61:50010 are bad. Aborting..." - The temp file is never closed even after the stream is undeployed or destroyed. Here are some details of our investigation that may be useful: - I start both the shell and the singlenode runner with --hadoopDistro phd1; I also configured the hadoop fs namenode correctly in the XD shell. - "http <options> | file <options>" work as expected; so does "http <options> | log" - "time | hdfs" does not show the same crash problem. Up until now only the http source combined with hdfs sink presents this problem - Putting a 4-10MB file in HDFS via the `Hadoop fs put` commands in Spring XD worked fine; so it's not a disk limitation. - This could be related to PHD service running on vCHS since supporting this configuration is fairly new. But it's only reproducable (consistently) with Spring XD's "http | hdfs" stream.

    Spring JIRA | 2 years ago | Sina Sojoodi
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.109.61:50010 remote=/192.168.109.11:46349]
  2. 0

    Problem syncing commit log: /hypertable/servers/rs18/log/user/3252: Error flushing DFS fd 73135

    Google Groups | 3 years ago | David
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.190.115.66:50010 remote=/10.190.115.56:52631]
  3. 0

    RegionServers Crashing every hour in production env

    Google Groups | 4 years ago | Pablo Musa
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.17.2.18:50010 remote=/172.17.2.18:37074]
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    reduce端缓存数据过多出现FGC,导致reduce生成的数据无法写到hdfs - 快乐程序员的专栏 - 博客频道 - CSDN.NET

    csdn.net | 1 year ago
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.39.5.193:50010 remote=/10.39.5.185:58225]
  6. 0

    Cloudera Manager Health Issue: NameNode Connectivity, Web Server Status

    Stack Overflow | 11 months ago | scott
    java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.248.200.105:50010 remote=/10.248.200.122:43572]

    2 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.net.SocketTimeoutException

      20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.109.61:50010 remote=/192.168.109.11:46349]

      at org.apache.hadoop.net.SocketIOWithTimeout.doIO()
    2. Hadoop
      SocketInputStream.read
      1. org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
      2. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
      3. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
      3 frames
    3. Java RT
      DataInputStream.read
      1. java.io.FilterInputStream.read(FilterInputStream.java:116)
      2. java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      3. java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      4. java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      5. java.io.DataInputStream.read(DataInputStream.java:132)
      5 frames
    4. Hadoop
      IOUtils.readFully
      1. org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
      1 frame
    5. Apache Hadoop HDFS
      DataXceiver.run
      1. org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
      2. org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
      3. org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
      4. org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:424)
      5. org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:656)
      6. org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:512)
      7. org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
      8. org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
      9. org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
      9 frames
    6. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:662)
      1 frame