java.net.SocketException: Broken pipe

grokbase.com | 3 months ago
  1. 0

    [ZooKeeper-user] New Instance can't sync - Grokbase

    grokbase.com | 3 months ago
    java.net.SocketException: Broken pipe
  2. 0

    The Zookeeper quorum, containing 5 servers, didn't provide service when restart after an "Out of memory" crash. It happened as following: 1. we built a Zookeeper quorum which contained 5 servers, say 1, 3, 4, 5, 6 (have no 2), and 6 was the leader. 2. we created 18 threads on 6 different servers to set and get data from a znode in the Zookeeper at the same time. The size of the data is 1MB. The test threads did their job as fast as possible, no pause between two operation, and they repeated the setting and getting 4000 times. 3. the Zookeeper leader crashed about 10 mins after the test threads started. The leader printed out the log: 2009-08-25 12:00:12,301 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x523 4223c2dc00b5 due to java.io.IOException: Read error 2009-08-25 12:00:12,318 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x523 4223c2dc00b6 due to java.io.IOException: Read error 2009-08-25 12:03:44,086 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x523 4223c2dc00b8 due to java.io.IOException: Read error 2009-08-25 12:04:53,757 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x523 4223c2dc00b7 due to java.io.IOException: Read error 2009-08-25 12:15:45,151 - FATAL [SyncThread:0:SyncRequestProcessor@131] - Severe unrecoverable error, exiting java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180) at org.apache.jute.BinaryOutputArchive.writeInt(BinaryOutputArchive.java:55) at org.apache.zookeeper.txn.SetDataTxn.serialize(SetDataTxn.java:42) at org.apache.zookeeper.server.persistence.Util.marshallTxnEntry(Util.java:262) at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:154) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:268) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:100) It is clear that the leader ran out of memory. then the server 4 was down almost at the same time, and printed out the log: 2009-08-25 12:15:45,995 - ERROR [FollowerRequestProcessor:3:FollowerRequestProcessor@91] - Unexpected exception causing exit java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) at java.io.DataOutputStream.write(DataOutputStream.java:90) at java.io.FilterOutputStream.write(FilterOutputStream.java:80) at org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119) at org.apache.zookeeper.server.quorum.QuorumPacket.serialize(QuorumPacket.java:51) at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123) at org.apache.zookeeper.server.quorum.Follower.writePacket(Follower.java:97) at org.apache.zookeeper.server.quorum.Follower.request(Follower.java:399) at org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:86) 2009-08-25 12:15:45,996 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x423 4ab894330075 due to java.net.SocketException: Broken pipe 2009-08-25 12:15:45,996 - FATAL [SyncThread:3:SyncRequestProcessor@131] - Severe unrecoverable error, exiting java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Follower.writePacket(Follower.java:100) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:52) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:147) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:92) 2009-08-25 12:15:45,995 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@309] - Exception when following the leader java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Follower.writePacket(Follower.java:100) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:256) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:498) 2009-08-25 12:15:46,022 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 12:15:46,022 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 12:15:46,023 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running It was really strange that when the 2 server down, the other three servers cannot provide service any more, the 'stat' command all leaded to "ZooKeeperServer not running". 4. I restarted the server 6(the former leader) and the server 4. But the service didn't come back. All the five servers printed "ZookeeperServer not running". The server 6 printed the logs: 2009-08-25 14:02:15,395 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:02:27,703 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: Responded to info probe 2009-08-25 14:02:28,733 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:02:42,070 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:02:55,407 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:03:08,744 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:03:22,080 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:03:29,396 - ERROR [main:Util@238] - Last transaction was partial. 2009-08-25 14:03:35,417 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:03:48,761 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running The server 4 printed logs like: 2009-08-25 14:03:48,747 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:04:02,091 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:04:15,427 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:04:17,816 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@164] - Unexpected exception, tries=0 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:525) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:156) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:498) 2009-08-25 14:04:18,820 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@164] - Unexpected exception, tries=1 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:525) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:156) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:498) 2009-08-25 14:04:19,823 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@164] - Unexpected exception, tries=2 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:525) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:156) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:498) 2009-08-25 14:04:28,764 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:04:42,101 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running the server 1, 3, 5 printed out the logs like: 2009-08-25 14:01:35,396 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: ZooKeeperServer not running 2009-08-25 14:01:36,554 - WARN [QuorumPeer:/0.0.0.0:2181:LeaderElection@194] - Ignoring exception while looking for lea der java.net.SocketTimeoutException: Receive timed out at java.net.PlainDatagramSocketImpl.receive0(Native Method) at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136) at java.net.DatagramSocket.receive(DatagramSocket.java:712) at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:170) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:488) 2009-08-25 14:01:37,758 - WARN [QuorumPeer:/0.0.0.0:2181:LeaderElection@194] - Ignoring exception while looking for lea der java.net.SocketTimeoutException: Receive timed out at java.net.PlainDatagramSocketImpl.receive0(Native Method) at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136) at java.net.DatagramSocket.receive(DatagramSocket.java:712) at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:170) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:488) 2009-08-25 14:01:37,865 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@164] - Unexpected exception, tries=0 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:525) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:156) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:498) 2009-08-25 14:01:38,289 - WARN [NIOServerCxn.Factory:2181:NIOServerCnxn@497] - Exception causing close of session 0x0 d ue to java.io.IOException: Responded to info probe my zoo.cfg is like: tickTime=2000 dataDir=./status/ clientPort=2181 initLimit=10 syncLimit=2 server.1=10.81.11.107:2888:3888 server.2=10.81.11.106:2888:3888 server.3=10.81.11.89:2888:3888 server.4=10.81.11.99:2888:3888 server.5=10.81.11.79:2888:3888 Several questions: 1. Why the leader selection failed after the restart? 2. Is the size of data too big to be processed properly? 3. How can I recover from this situation? Can I just remove the version-2 directory on server 6(the former leader) and restart the server?

    Apache's JIRA Issue Tracker | 7 years ago | Qian Ye
    java.net.SocketException: Broken pipe
  3. 0

    [ZOOKEEPER-1115] follower can not sync with leader - ASF JIRA

    apache.org | 12 months ago
    java.net.SocketException: Socket closed
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hi, I need some help understanding a recurring problem we're seeing with our zookeeper cluster. It's a five node cluster that ordinarily runs fine. Occasionally we see an error from which the cluster recovers, but it causes a lot of grief and I'm sure is representative of an unhealthy situation. To my eye it looks like an invalid bit of data getting into the system and not being handled gracefully; I'm the first to say my eye is not expert though, so I humbly submit an annotated log exert in the hope some who knows more than me can provide some illumination. The cluster seems to be ticking along fine, until we get errors on 2 of the 5 nodes like so: 2016-01-19 13:12:49,698 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@89] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786) 2016-01-19 13:12:49,698 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790) This is immediately followed by 380 occurences of: 2016-01-19 13:12:49,699 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.56:59028 which had sessionid 0x151b01ee8330234 and a: 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerZooKeeperServer@139] - Shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@441] - shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerRequestProcessor@105] - Shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:CommitProcessor@181] - Shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FinalRequestProcessor@415] - shutdown of request processor complete 2016-01-19 13:12:49,767 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:SyncRequestProcessor@209] - Shutting down 2016-01-19 13:12:49,767 - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop! 2016-01-19 13:12:49,767 - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop! 2016-01-19 13:13:09,418 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 30334ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2016-01-19 13:13:09,427 - WARN [SyncThread:1:SendAckRequestProcessor@64] - Closing connection to leader, exception during packet send java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:139) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:62) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:204) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131) 2016-01-19 13:13:09,428 - INFO [SyncThread:1:SyncRequestProcessor@187] - SyncRequestProcessor exited! As a small aside, the fsync log errors for the first two servers to be hit are: 2016-01-19 13:13:09,418 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 30334ms which will adversely effect operation latency. 2016-01-19 13:13:09,539 - WARN [SyncThread:2:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:2 took 30456ms which will adversely effect operation latency. If rewind from date of the entry the milliseconds given, you arrive within one millisecond of the same time on each server. But I digress. For the next 12 minutes or so, the logs are full of the below sort of exceptions, in seemingly no consistent order or frequency: 2016-01-19 13:13:09,440 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2016-01-19 13:13:09,441 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.181:51381 (no session established for client) 2016-01-19 13:13:09,443 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:SendAckRequestProcessor@64] - Closing connection to leader, exception during packet send java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:139) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:62) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:204) at org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:216) at org.apache.zookeeper.server.quorum.FollowerZooKeeperServer.shutdown(FollowerZooKeeperServer.java:147) at org.apache.zookeeper.server.quorum.Learner.shutdown(Learner.java:546) at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790) 2016-01-19 13:13:09,443 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING 2016-01-19 13:13:11,782 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1525a047dc20005, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:744) 2016-01-19 13:13:11,783 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.1:59576 which had sessionid 0x1525a047dc20005 2016-01-19 13:13:11,784 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2016-01-19 13:25:43,898 - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x2a001d352d (n.zxid), 0xb (n.round), LOOKING (n.state), 2 (n.sid), 0x2a (n.peerEpoch) FOLLOWING (my state) 2016-01-19 13:25:43,901 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@89] - Exception when following the leader java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786) 2016-01-19 13:25:43,901 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790) Until eventually we get to: 2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@784] - FOLLOWING 2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@162] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper_1/data/version-2 snapdir /var/lib/zookeeper_1/data/version-2 2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 21179 2016-01-19 13:26:05,100 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Learner@233] - Unexpected exception, tries=0, connecting to zoo005/X.Y.Z.71:2888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:225) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:71) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786) And things start to come right. Right about now, the three member which had so far escaped begin to exhibit the same behaviour. Again, if we look at the fsync messages: 2016-01-19 13:26:06,192 - WARN [SyncThread:3:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:3 took 51394ms which will adversely effect operation latency. 2016-01-19 13:26:05,960 - WARN [SyncThread:4:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:4 took 51162ms which will adversely effect operation latency. 2016-01-19 13:26:04,524 - WARN [SyncThread:5:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:5 took 49726ms which will adversely effect operation latency. And we rewind the number of milliseconds from the log entry timestamp, we arrive at exactly 13:25:14,798 for all three events. So, it looks for all the world like something entered the system at 13:12:39,084, caused havoc on two nodes for 12.5 minutes, then at 13:25:14,798 it got off those and made to the other three, where it again caused havoc, before things eventually recovered and the world kept on ticking, only a medium sized log explosion worse for it. There is nothing in any of the logs within a second of either of those times. I'm hoping someone familiar with the code can look at those stack traces and understand what might cause such an incident. I'm to help anyway I can. I have more complete logs, and we see this every couple of weeks or so, so can setup some additional logging if it would be of value. Let me know.

    Apache's JIRA Issue Tracker | 11 months ago | Gareth Humphries
    java.net.SocketException: Socket closed
  6. 0

    exception causing shutdownthere are 5 members in the quorum. one follower can not sync with leader after restart. it seems leader has close the data connection with this follower because of read timeout. here is the key log in follower: {noformat} 2011-06-30 22:14:45,069 - WARN [Thread-17:QuorumCnxManager$RecvWorker@658] - Connection broken: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:113) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:156) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) 2011-06-30 22:14:45,069 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@689] - Notification: 3, 17198470148, 3, 3, LOOKING, LOOKING, 3 2011-06-30 22:14:45,070 - ERROR [Thread-16:QuorumCnxManager$SendWorker@559] - Failed to send last message. Shutting down thread. java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:557) 2011-06-30 22:14:45,082 - INFO [QuorumPeer:/0.0.0.0:2181:Learner@282] - Getting a diff from the leader 0x4011bd462 2011-06-30 22:14:45,083 - WARN [Thread-18:QuorumCnxManager$SendWorker@589] - Send worker leaving thread 2011-06-30 22:14:45,085 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@116] - Got zxid 0x4011bd405 expected 0x1 2011-06-30 22:14:45,090 - INFO [QuorumPeer:/0.0.0.0:2181:FileTxnSnapLog@208] - Snapshotting: 4011bd462 2011-06-30 22:14:53,397 - WARN [SyncThread:3:SendAckRequestProcessor@63] - Closing connection to leader, exception during packet send java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:126) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:61) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:164) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98) 2011-06-30 22:14:53,398 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] - Exception when following the leader java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:126) at org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:358) at org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:108) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:634) 2011-06-30 22:14:53,398 - WARN [SyncThread:3:SendAckRequestProcessor@63] - Closing connection to leader, exception during packet send java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:126) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:61) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:164) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98) 2011-06-30 22:14:53,399 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:638) and these are the leader's: 2011-06-30 22:14:35,943 - ERROR [LearnerHandler-/10.23.247.163:14975:LearnerHandler@444] - Unexpected exception causing shutdown while sock still open java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:358) 2011-06-30 22:14:35,943 - WARN [LearnerHandler-/10.23.247.163:14975:LearnerHandler@457] - ******* GOODBYE /10.23.247.163:14975 ******** 2011-06-30 22:14:48,943 - ERROR [CommitProcessor:4:NIOServerCnxn@422] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:395) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1360) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:367) at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:535) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) 2011-06-30 22:14:49,084 - ERROR [LearnerHandler-/10.23.247.163:14998:LearnerHandler@444] - Unexpected exception causing shutdown while sock still open java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:358) 2011-06-30 22:14:49,084 - WARN [LearnerHandler-/10.23.247.163:14998:LearnerHandler@457] - ******* GOODBYE /10.23.247.163:14998 ******** {noformat}

    Apache's JIRA Issue Tracker | 5 years ago | helei
    java.net.SocketException: Socket closed

  1. kuldeep 4 times, last 2 months ago
  2. esaar 1 times, last 4 months ago
  3. mauritius 2 times, last 9 months ago
16 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.net.SocketException

    Broken pipe

    at java.net.SocketOutputStream.socketWrite0()
  2. Java RT
    BufferedOutputStream.flush
    1. java.net.SocketOutputStream.socketWrite0(Native Method)
    2. java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
    3. java.net.SocketOutputStream.write(SocketOutputStream.java:153)
    4. java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    5. java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    5 frames
  3. Zookeeper
    QuorumPeer.run
    1. org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:134)
    2. org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:418)
    3. org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:108)
    4. org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79)
    5. org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
    5 frames