java.io.EOFException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • [ZooKeeper-user] New Instance can't sync - Grokbase
    via by Unknown author,
  • Exception when following leader
    via by Unknown author,
  • ZooKeeper - SOLR issue
    via Stack Overflow by J-S
    ,
  • Hi, I need some help understanding a recurring problem we're seeing with our zookeeper cluster. It's a five node cluster that ordinarily runs fine. Occasionally we see an error from which the cluster recovers, but it causes a lot of grief and I'm sure is representative of an unhealthy situation. To my eye it looks like an invalid bit of data getting into the system and not being handled gracefully; I'm the first to say my eye is not expert though, so I humbly submit an annotated log exert in the hope some who knows more than me can provide some illumination. The cluster seems to be ticking along fine, until we get errors on 2 of the 5 nodes like so: 2016-01-19 13:12:49,698 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@89] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786) 2016-01-19 13:12:49,698 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790) This is immediately followed by 380 occurences of: 2016-01-19 13:12:49,699 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.56:59028 which had sessionid 0x151b01ee8330234 and a: 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerZooKeeperServer@139] - Shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@441] - shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerRequestProcessor@105] - Shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:CommitProcessor@181] - Shutting down 2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FinalRequestProcessor@415] - shutdown of request processor complete 2016-01-19 13:12:49,767 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:SyncRequestProcessor@209] - Shutting down 2016-01-19 13:12:49,767 - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop! 2016-01-19 13:12:49,767 - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop! 2016-01-19 13:13:09,418 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 30334ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2016-01-19 13:13:09,427 - WARN [SyncThread:1:SendAckRequestProcessor@64] - Closing connection to leader, exception during packet send java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:139) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:62) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:204) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131) 2016-01-19 13:13:09,428 - INFO [SyncThread:1:SyncRequestProcessor@187] - SyncRequestProcessor exited! As a small aside, the fsync log errors for the first two servers to be hit are: 2016-01-19 13:13:09,418 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 30334ms which will adversely effect operation latency. 2016-01-19 13:13:09,539 - WARN [SyncThread:2:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:2 took 30456ms which will adversely effect operation latency. If rewind from date of the entry the milliseconds given, you arrive within one millisecond of the same time on each server. But I digress. For the next 12 minutes or so, the logs are full of the below sort of exceptions, in seemingly no consistent order or frequency: 2016-01-19 13:13:09,440 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2016-01-19 13:13:09,441 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.181:51381 (no session established for client) 2016-01-19 13:13:09,443 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:SendAckRequestProcessor@64] - Closing connection to leader, exception during packet send java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:139) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:62) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:204) at org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:216) at org.apache.zookeeper.server.quorum.FollowerZooKeeperServer.shutdown(FollowerZooKeeperServer.java:147) at org.apache.zookeeper.server.quorum.Learner.shutdown(Learner.java:546) at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790) 2016-01-19 13:13:09,443 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING 2016-01-19 13:13:11,782 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1525a047dc20005, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:744) 2016-01-19 13:13:11,783 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.1:59576 which had sessionid 0x1525a047dc20005 2016-01-19 13:13:11,784 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2016-01-19 13:25:43,898 - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x2a001d352d (n.zxid), 0xb (n.round), LOOKING (n.state), 2 (n.sid), 0x2a (n.peerEpoch) FOLLOWING (my state) 2016-01-19 13:25:43,901 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@89] - Exception when following the leader java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786) 2016-01-19 13:25:43,901 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790) Until eventually we get to: 2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@784] - FOLLOWING 2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@162] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper_1/data/version-2 snapdir /var/lib/zookeeper_1/data/version-2 2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 21179 2016-01-19 13:26:05,100 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Learner@233] - Unexpected exception, tries=0, connecting to zoo005/X.Y.Z.71:2888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:225) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:71) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786) And things start to come right. Right about now, the three member which had so far escaped begin to exhibit the same behaviour. Again, if we look at the fsync messages: 2016-01-19 13:26:06,192 - WARN [SyncThread:3:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:3 took 51394ms which will adversely effect operation latency. 2016-01-19 13:26:05,960 - WARN [SyncThread:4:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:4 took 51162ms which will adversely effect operation latency. 2016-01-19 13:26:04,524 - WARN [SyncThread:5:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:5 took 49726ms which will adversely effect operation latency. And we rewind the number of milliseconds from the log entry timestamp, we arrive at exactly 13:25:14,798 for all three events. So, it looks for all the world like something entered the system at 13:12:39,084, caused havoc on two nodes for 12.5 minutes, then at 13:25:14,798 it got off those and made to the other three, where it again caused havoc, before things eventually recovered and the world kept on ticking, only a medium sized log explosion worse for it. There is nothing in any of the logs within a second of either of those times. I'm hoping someone familiar with the code can look at those stack traces and understand what might cause such an incident. I'm to help anyway I can. I have more complete logs, and we see this every couple of weeks or so, so can setup some additional logging if it would be of value. Let me know.
    via by Gareth Humphries,
  • CnxManagerTest.testWorkerThreads See attachment, this is the first time I've seen this test fail, and it's failed 2 out of the last three test runs. Notice (attachment) once this happens the port never becomes available. {noformat} 2011-03-02 15:53:12,425 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxn$Factory@251] - Accepted socket connection from /172.29.6.162:51441 2011-03-02 15:53:12,430 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxn@639] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2011-03-02 15:53:12,430 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxn@1435] - Closed socket connection for client /172.29.6.162:51441 (no session established for client) 2011-03-02 15:53:12,430 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:11241:Follower@82] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148) at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:267) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:66) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645) 2011-03-02 15:53:12,431 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11241:Follower@165] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649) 2011-03-02 15:53:12,432 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11241:QuorumPeer@621] - LOOKING 2011-03-02 15:53:12,432 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11241:FastLeaderElection@663] - New election. My id = 0, Proposed zxid = 0 2011-03-02 15:53:12,433 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 0 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state) 2011-03-02 15:53:12,433 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 0 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state) 2011-03-02 15:53:12,433 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 0 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state) 2011-03-02 15:53:12,633 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 0 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state) 2011-03-02 15:53:12,633 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11245:QuorumPeer@655] - LEADING 2011-03-02 15:53:12,636 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11245:Leader@54] - TCP NoDelay set to: true 2011-03-02 15:53:12,638 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11245:ZooKeeperServer@151] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /var/lib/hudson/workspace/CDH3-ZooKeeper-3.3.3_sles/build/test/tmp/test9001250572426375869.junit.dir/version-2 snapdir /var/lib/hudson/workspace/CDH3-ZooKeeper-3.3.3_sles/build/test/tmp/test9001250572426375869.junit.dir/version-2 2011-03-02 15:53:12,639 - ERROR [QuorumPeer:/0:0:0:0:0:0:0:0:11245:Leader@133] - Couldn't bind to port 11245 java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365) at java.net.ServerSocket.bind(ServerSocket.java:319) at java.net.ServerSocket.<init>(ServerSocket.java:185) at java.net.ServerSocket.<init>(ServerSocket.java:97) at org.apache.zookeeper.server.quorum.Leader.<init>(Leader.java:131) at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:512) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:657) {noformat}
    via by Patrick Hunt,
  • here are the part of the log whereby my zookeeper instance crashed, taking 3 out of 5 down, and thus ruining the quorum for all clients: 2009-07-23 12:29:06,769 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x52276d1d5161350 due to java.io.IOException: Read error 2009-07-23 12:29:00,756 WARN org.apache.zookeeper.server.quorum.Follower: Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:65) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.Follower.readPacket(Follower.java:114) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:243) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:494) 2009-07-23 12:29:06,770 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5161350 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.168:39489] 2009-07-23 12:29:06,770 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb0578 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.159:46797] 2009-07-23 12:29:06,771 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa013e NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.153:33998] 2009-07-23 12:29:06,771 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x52276d1d5160593 due to java.io.IOException: Read error 2009-07-23 12:29:06,808 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e02bb NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.158:53758] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa13e4 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.154:58681] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691382 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59967] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb1354 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.163:49957] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa13cd NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.150:34212] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691383 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.159:46813] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb0350 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59956] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e139b NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.156:55138] 2009-07-23 12:29:06,809 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e1398 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.167:41257] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5161355 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.153:34032] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d516011c NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.155:56314] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e056b NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.155:56322] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d516011f NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.157:49618] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e11ea NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.10.20.42:55483] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e02ba NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.157:49632] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb1355 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.169:58824] 2009-07-23 12:29:06,810 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691378 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.161:40973] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691380 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59944] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e0311 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.160:56167] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e690374 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.169:58815] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e139f NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.151:51396] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e139c NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.155:56315] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e69137b NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59859] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5160594 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.151:51370] 2009-07-23 12:29:06,811 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e69137a NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.159:46682] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5160347 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.165:35722] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e69137f NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.159:46754] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5160121 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.155:56307] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb0126 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.154:58688] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa05fc NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.152:45067] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e0316 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.169:58800] 2009-07-23 12:29:06,812 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e69137e NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.159:46737] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e69137d NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.159:46733] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa13df NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.156:55137] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb134e NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.166:40443] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691381 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.161:41086] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5161356 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.165:35719] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb1349 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10. 20.20.158:53770] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x12276d15dfb0352 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.165:35718] 2009-07-23 12:29:06,813 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691379 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59823] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d516000e NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.150:34216] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x32276d15d2e1397 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.169:58829] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e69137c NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59862] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa0140 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.155:56271] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x42276d1d3fa13e1 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.157:49608] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x22276d15e691377 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.162:59789] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x52276d1d5160593 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.165:35703] 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request processor complete 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.quorum.FollowerRequestProcessor: FollowerRequestProcessor exited loop! 2009-07-23 12:29:06,814 INFO org.apache.zookeeper.server.quorum.CommitProcessor: CommitProcessor exited loop! 2009-07-23 12:29:06,815 INFO org.apache.zookeeper.server.quorum.Follower: shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:427) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:498) 2009-07-23 12:29:06,815 WARN org.apache.zookeeper.server.NIOServerCnxn: Ignoring exception java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55) at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:201) 2009-07-23 12:29:06,815 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING 2009-07-23 12:29:06,817 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2009-07-23 12:29:06,817 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.156:55206] 2009-07-23 12:29:06,818 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2009-07-23 12:29:06,818 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.155:56331] [elided lots of the same] 2009-07-23 12:29:33,008 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/10.20.20.151:2181 remote=/10.20.20.152:5945 8] 2009-07-23 12:29:33,011 FATAL org.apache.zookeeper.server.SyncRequestProcessor: Severe unrecoverable error, exiting java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Follower.writePacket(Follower.java:100) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:52) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:131) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:76) The good news is when I restarted the downed zookeepers, everything returned to normal.
    via by ryan rawson,
  • {noformat} Error Message Broken pipe Stacktrace java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:98) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at java.io.OutputStream.write(OutputStream.java:75) at java.nio.channels.Channels$1.write(Channels.java:155) at java.io.DataOutputStream.writeInt(DataOutputStream.java:198) at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion(CnxManagerTest.java:318) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) Standard Output 2016-07-12 22:34:46,623 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxFromFutureVersion 2016-07-12 22:34:46,627 [myid:] - INFO [main:PortAssignment@32] - assigning port 11221 2016-07-12 22:34:46,630 [myid:] - INFO [main:PortAssignment@32] - assigning port 11222 2016-07-12 22:34:46,631 [myid:] - INFO [main:PortAssignment@32] - assigning port 11223 2016-07-12 22:34:46,643 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:46,658 [myid:] - INFO [main:PortAssignment@32] - assigning port 11224 2016-07-12 22:34:46,658 [myid:] - INFO [main:PortAssignment@32] - assigning port 11225 2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11226 2016-07-12 22:34:46,659 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11227 2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11228 2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11229 2016-07-12 22:34:46,660 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:46,660 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxFromFutureVersion 2016-07-12 22:34:46,672 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11225 2016-07-12 22:34:46,692 [myid:] - INFO [main:CnxManagerTest@301] - Election port: 11226 2016-07-12 22:34:46,692 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11226 2016-07-12 22:34:47,696 [myid:] - INFO [/0.0.0.0:11226:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:48724 2016-07-12 22:34:49,700 [myid:] - WARN [/0.0.0.0:11226:QuorumCnxManager@274] - Exception reading or writing challenge: java.net.SocketTimeoutException: Read timed out 2016-07-12 22:34:52,700 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@74] - TEST METHOD FAILED testCnxFromFutureVersion java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:98) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at java.io.OutputStream.write(OutputStream.java:75) at java.nio.channels.Channels$1.write(Channels.java:155) at java.io.DataOutputStream.writeInt(DataOutputStream.java:198) at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion(CnxManagerTest.java:318) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:34:52,705 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testCnxFromFutureVersion java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:98) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at java.io.OutputStream.write(OutputStream.java:75) at java.nio.channels.Channels$1.write(Channels.java:155) at java.io.DataOutputStream.writeInt(DataOutputStream.java:198) at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion(CnxManagerTest.java:318) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:34:52,706 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxFromFutureVersion 2016-07-12 22:34:52,720 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testSocketTimeout 2016-07-12 22:34:52,720 [myid:] - INFO [main:PortAssignment@32] - assigning port 11230 2016-07-12 22:34:52,720 [myid:] - INFO [main:PortAssignment@32] - assigning port 11231 2016-07-12 22:34:52,720 [myid:] - INFO [main:PortAssignment@32] - assigning port 11232 2016-07-12 22:34:52,721 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:52,721 [myid:] - INFO [main:PortAssignment@32] - assigning port 11233 2016-07-12 22:34:52,722 [myid:] - INFO [main:PortAssignment@32] - assigning port 11234 2016-07-12 22:34:52,722 [myid:] - INFO [main:PortAssignment@32] - assigning port 11235 2016-07-12 22:34:52,722 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:52,723 [myid:] - INFO [main:PortAssignment@32] - assigning port 11236 2016-07-12 22:34:52,723 [myid:] - INFO [main:PortAssignment@32] - assigning port 11237 2016-07-12 22:34:52,723 [myid:] - INFO [main:PortAssignment@32] - assigning port 11238 2016-07-12 22:34:52,724 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:52,724 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testSocketTimeout 2016-07-12 22:34:52,725 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11234 2016-07-12 22:34:52,726 [myid:] - INFO [main:CnxManagerTest@370] - Election port: 11235 2016-07-12 22:34:52,726 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11235 2016-07-12 22:34:53,729 [myid:] - INFO [/0.0.0.0:11235:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:34496 2016-07-12 22:34:57,734 [myid:] - WARN [/0.0.0.0:11235:QuorumCnxManager@274] - Exception reading or writing challenge: java.net.SocketTimeoutException: Read timed out 2016-07-12 22:34:57,734 [myid:] - WARN [main:QuorumCnxManager@274] - Exception reading or writing challenge: java.io.EOFException 2016-07-12 22:34:57,734 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 3153 2016-07-12 22:34:57,735 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 6 2016-07-12 22:34:57,735 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testSocketTimeout 2016-07-12 22:34:57,735 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testSocketTimeout 2016-07-12 22:34:57,735 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testSocketTimeout 2016-07-12 22:34:57,735 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testWorkerThreads 2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11239 2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11240 2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11241 2016-07-12 22:34:57,736 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11242 2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11243 2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11244 2016-07-12 22:34:57,737 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11245 2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11246 2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11247 2016-07-12 22:34:57,737 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:57,738 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testWorkerThreads 2016-07-12 22:34:57,738 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11240 2016-07-12 22:34:57,784 [myid:] - INFO [main:CnxManagerTest@393] - Starting peer 0 2016-07-12 22:34:57,789 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-12 22:34:57,905 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-12 22:34:57,945 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11241 2016-07-12 22:34:57,953 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11243 2016-07-12 22:34:57,954 [myid:] - INFO [main:CnxManagerTest@393] - Starting peer 1 2016-07-12 22:34:57,954 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-12 22:34:58,024 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@774] - LOOKING 2016-07-12 22:34:58,025 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-12 22:34:58,025 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FastLeaderElection@818] - New election. My id = 0, proposed zxid=0x0 2016-07-12 22:34:58,028 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,028 [myid:] - WARN [WorkerSender[myid=0]:QuorumCnxManager@400] - Cannot open channel to 1 at election address /0.0.0.0:11244 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:744) 2016-07-12 22:34:58,029 [myid:] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:58,030 [myid:] - WARN [WorkerSender[myid=0]:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:11247 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:744) 2016-07-12 22:34:58,030 [myid:] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:58,050 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11244 2016-07-12 22:34:58,052 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11246 2016-07-12 22:34:58,052 [myid:] - INFO [main:CnxManagerTest@393] - Starting peer 2 2016-07-12 22:34:58,053 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-12 22:34:58,057 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@774] - LOOKING 2016-07-12 22:34:58,058 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:FastLeaderElection@818] - New election. My id = 1, proposed zxid=0x0 2016-07-12 22:34:58,058 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46871 2016-07-12 22:34:58,063 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,064 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,064 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,065 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,065 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,065 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:11247 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:744) 2016-07-12 22:34:58,065 [myid:] - WARN [WorkerSender[myid=0]:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:11247 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:744) 2016-07-12 22:34:58,066 [myid:] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:58,066 [myid:] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:58,084 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-12 22:34:58,133 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11247 2016-07-12 22:34:58,136 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer@774] - LOOKING 2016-07-12 22:34:58,136 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FastLeaderElection@818] - New election. My id = 2, proposed zxid=0x0 2016-07-12 22:34:58,137 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46874 2016-07-12 22:34:58,138 [myid:] - INFO [/0.0.0.0:11244:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:33135 2016-07-12 22:34:58,141 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,143 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,143 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,143 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,145 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,145 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:58,345 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@844] - FOLLOWING 2016-07-12 22:34:58,345 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@844] - FOLLOWING 2016-07-12 22:34:58,346 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer@856] - LEADING 2016-07-12 22:34:58,349 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Learner@86] - TCP NoDelay set to: true 2016-07-12 22:34:58,350 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Leader@59] - TCP NoDelay set to: true 2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:zookeeper.version=3.4.9-SNAPSHOT-1752356, built on 07/12/2016 22:30 GMT 2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:host.name=hemera.apache.org 2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.version=1.8.0 2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.vendor=Oracle Corporation 2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.home=/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.8.0/jre 2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.class.path=/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/classes:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/antlr-2.7.6.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/checkstyle-5.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-cli-1.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-lang-1.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/google-collections-0.9.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/junit-4.8.1.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/classes:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/jline-0.9.94.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/log4j-1.2.16.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/netty-3.10.5.Final.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/clover/latest/lib/clover.jar:/x1/jenkins/tools/ant/apache-ant-1.8.2/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.io.tmpdir=/tmp 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.compiler=<NA> 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:os.name=Linux 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:os.arch=amd64 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:os.version=3.2.0-104-generic 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:user.name=jenkins 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:user.home=/home/jenkins 2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:user.dir=/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4 2016-07-12 22:34:58,358 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test6977370939981061612.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test6977370939981061612.junit.dir/version-2 2016-07-12 22:34:58,358 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2 2016-07-12 22:34:58,358 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2 2016-07-12 22:34:58,359 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 301 2016-07-12 22:34:58,359 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 334 2016-07-12 22:34:58,365 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Leader@361] - LEADING - LEADER ELECTION TOOK - 224 2016-07-12 22:34:58,370 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:58,373 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:58,383 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@329] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@4e46c644 2016-07-12 22:34:58,383 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@329] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@23da3d 2016-07-12 22:34:58,528 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@384] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0 2016-07-12 22:34:58,528 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@384] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0 2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@393] - leader and follower are in sync, zxid=0x0 2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@393] - leader and follower are in sync, zxid=0x0 2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@458] - Sending DIFF 2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@458] - Sending DIFF 2016-07-12 22:34:58,530 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Learner@326] - Getting a diff from the leader 0x0 2016-07-12 22:34:58,530 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Learner@326] - Getting a diff from the leader 0x0 2016-07-12 22:34:58,534 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:FileTxnSnapLog@240] - Snapshotting: 0x0 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test6977370939981061612.junit.dir/version-2/snapshot.0 2016-07-12 22:34:58,534 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FileTxnSnapLog@240] - Snapshotting: 0x0 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2/snapshot.0 2016-07-12 22:34:58,623 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@518] - Received NEWLEADER-ACK message from 0 2016-07-12 22:34:58,623 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@518] - Received NEWLEADER-ACK message from 1 2016-07-12 22:34:58,623 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Leader@946] - Have quorum of supporters, sids: [ 0,2 ]; starting up and setting last processed zxid: 0x100000000 2016-07-12 22:34:58,634 [myid:] - INFO [main:CnxManagerTest@405] - Round 0, halting peer 0 2016-07-12 22:34:58,634 [myid:] - INFO [main:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891) at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:407) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:34:58,635 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:34:58,645 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11240:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-12 22:34:58,645 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:34:58,647 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-12 22:34:58,647 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-12 22:34:58,648 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down 2016-07-12 22:34:58,648 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:34:58,648 [myid:] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop! 2016-07-12 22:34:58,648 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-12 22:34:58,649 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-12 22:34:58,649 [myid:] - INFO [CommitProcessor:0:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-12 22:34:58,650 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:34:58,650 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:34:58,651 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-12 22:34:58,651 [myid:] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-12 22:34:58,651 [myid:] - ERROR [/0.0.0.0:11241:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-12 22:34:58,652 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:34:58,653 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 0, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:34:58,654 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:34:58,653 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 2, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:34:58,653 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:34:58,656 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:34:58,652 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 1, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:34:58,656 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:34:58,652 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 0, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:34:58,657 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:34:58,657 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:34:58,658 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:34:58,655 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:34:58,654 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:34:58,659 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:34:58,659 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:34:59,154 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11240 2016-07-12 22:34:59,155 [myid:] - INFO [main:CnxManagerTest@414] - Round {}, restarting peer {}[Ljava.lang.Object;@12bc6874 2016-07-12 22:34:59,157 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2/snapshot.0 2016-07-12 22:34:59,159 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11241 2016-07-12 22:34:59,161 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@774] - LOOKING 2016-07-12 22:34:59,162 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FastLeaderElection@818] - New election. My id = 0, proposed zxid=0x0 2016-07-12 22:34:59,162 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:59,163 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (1, 0) 2016-07-12 22:34:59,163 [myid:] - INFO [/0.0.0.0:11244:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:33142 2016-07-12 22:34:59,163 [myid:] - INFO [/0.0.0.0:11247:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:44785 2016-07-12 22:34:59,163 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (2, 0) 2016-07-12 22:34:59,164 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46884 2016-07-12 22:34:59,166 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46885 2016-07-12 22:34:59,167 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEpoch) FOLLOWING (my state) 2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEpoch) LEADING (my state) 2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:59,170 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LEADING (n.state), 2 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state) 2016-07-12 22:34:59,170 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@844] - FOLLOWING 2016-07-12 22:34:59,170 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2 2016-07-12 22:34:59,170 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 8 2016-07-12 22:34:59,171 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@329] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@23da3d 2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@384] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0 2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@458] - Sending SNAP 2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@482] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x100000000 2016-07-12 22:34:59,172 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Learner@329] - Getting a snapshot from leader 2016-07-12 22:34:59,174 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FileTxnSnapLog@240] - Snapshotting: 0x100000000 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2/snapshot.100000000 2016-07-12 22:34:59,284 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@518] - Received NEWLEADER-ACK message from 0 2016-07-12 22:34:59,652 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-12 22:34:59,660 [myid:] - INFO [main:CnxManagerTest@405] - Round 1, halting peer 0 2016-07-12 22:34:59,660 [myid:] - INFO [main:Follower@166] - shutdown called j ...[truncated 254755 chars]... Keeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2 2016-07-12 22:35:13,328 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 3 2016-07-12 22:35:13,329 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,329 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@329] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@6f23fc6d 2016-07-12 22:35:13,330 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@384] - Synchronizing with Follower sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x200000000 2016-07-12 22:35:13,330 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@393] - leader and follower are in sync, zxid=0x200000000 2016-07-12 22:35:13,330 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@458] - Sending DIFF 2016-07-12 22:35:13,330 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Learner@326] - Getting a diff from the leader 0x200000000 2016-07-12 22:35:13,331 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FileTxnSnapLog@240] - Snapshotting: 0x200000000 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2/snapshot.200000000 2016-07-12 22:35:13,376 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@518] - Received NEWLEADER-ACK message from 2 2016-07-12 22:35:13,821 [myid:] - INFO [/0.0.0.0:11247:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-12 22:35:13,824 [myid:] - INFO [main:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891) at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:424) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:35:13,824 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:35:13,824 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-12 22:35:13,825 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down 2016-07-12 22:35:13,825 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-12 22:35:13,825 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-12 22:35:13,825 [myid:] - INFO [CommitProcessor:0:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-12 22:35:13,826 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:35:13,825 [myid:] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop! 2016-07-12 22:35:13,826 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-12 22:35:13,827 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11240:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-12 22:35:13,828 [myid:] - ERROR [/0.0.0.0:11241:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-12 22:35:13,829 [myid:] - INFO [main:Leader@496] - Shutting down 2016-07-12 22:35:13,828 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 1, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,829 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,828 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 2, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,829 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,828 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,828 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,828 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 0, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,830 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,829 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,831 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,829 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 0, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,831 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,829 [myid:] - INFO [main:Leader@502] - Shutdown called java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888) at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:424) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:35:13,831 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-12 22:35:13,831 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down 2016-07-12 22:35:13,832 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down 2016-07-12 22:35:13,832 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down 2016-07-12 22:35:13,832 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-12 22:35:13,832 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down 2016-07-12 22:35:13,832 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-12 22:35:13,832 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:35:13,831 [myid:] - INFO [LearnerCnxAcceptor-/0.0.0.0:11242:Leader$LearnerCnxAcceptor@325] - exception while shutting down acceptor: java.net.SocketException: Socket closed 2016-07-12 22:35:13,832 [myid:] - INFO [SyncThread:1:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-12 22:35:13,832 [myid:] - INFO [CommitProcessor:1:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-12 22:35:13,832 [myid:] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! 2016-07-12 22:35:13,834 [myid:] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Follower@89] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846) 2016-07-12 22:35:13,833 [myid:] - WARN [LearnerHandler-/140.211.11.27:32993:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32993 ******** 2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32990:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32990 ******** 2016-07-12 22:35:13,834 [myid:] - WARN [LearnerHandler-/140.211.11.27:32980:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32980 ******** 2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32980:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-12 22:35:13,834 [myid:] - WARN [LearnerHandler-/140.211.11.27:32997:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32997 ******** 2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32997:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-12 22:35:13,835 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11243:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32990:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-12 22:35:13,836 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@862] - Unexpected exception java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859) 2016-07-12 22:35:13,836 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Leader@496] - Shutting down 2016-07-12 22:35:13,836 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-12 22:35:13,835 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:ZooKeeperServer@469] - shutting down 2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FollowerRequestProcessor@107] - Shutting down 2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32993:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-12 22:35:13,835 [myid:] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@89] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846) 2016-07-12 22:35:13,837 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop! 2016-07-12 22:35:13,838 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:CommitProcessor@184] - Shutting down 2016-07-12 22:35:13,839 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-12 22:35:13,839 [myid:] - INFO [CommitProcessor:2:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-12 22:35:13,837 [myid:] - INFO [main:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891) at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:424) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:35:13,839 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:35:13,839 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:35:13,836 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 1, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,840 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,836 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,840 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,836 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,840 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,836 [myid:] - ERROR [/0.0.0.0:11244:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-12 22:35:13,840 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,841 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,840 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-12 22:35:13,839 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:35:13,841 [myid:] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-12 22:35:13,841 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11246:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-12 22:35:13,839 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FollowerZooKeeperServer@140] - Shutting down 2016-07-12 22:35:13,842 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:SyncRequestProcessor@209] - Shutting down 2016-07-12 22:35:13,842 [myid:] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-12 22:35:13,842 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 11673 2016-07-12 22:35:13,842 [myid:] - ERROR [/0.0.0.0:11247:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-12 22:35:13,842 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 2016-07-12 22:35:13,842 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testWorkerThreads 2016-07-12 22:35:13,842 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testWorkerThreads 2016-07-12 22:35:13,843 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testWorkerThreads 2016-07-12 22:35:13,843 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxManager 2016-07-12 22:35:13,843 [myid:] - INFO [main:PortAssignment@32] - assigning port 11248 2016-07-12 22:35:13,843 [myid:] - INFO [main:PortAssignment@32] - assigning port 11249 2016-07-12 22:35:13,843 [myid:] - INFO [main:PortAssignment@32] - assigning port 11250 2016-07-12 22:35:13,843 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11251 2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11252 2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11253 2016-07-12 22:35:13,844 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11254 2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11255 2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11256 2016-07-12 22:35:13,844 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,844 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxManager 2016-07-12 22:35:13,845 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11252 2016-07-12 22:35:13,846 [myid:] - INFO [Thread-19:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11249 2016-07-12 22:35:13,846 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11253 2016-07-12 22:35:13,846 [myid:] - WARN [main:QuorumCnxManager@400] - Cannot open channel to 0 at election address /0.0.0.0:11250 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxManager(CnxManagerTest.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:35:13,847 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,847 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11250 2016-07-12 22:35:13,847 [myid:] - INFO [/0.0.0.0:11253:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:41746 2016-07-12 22:35:13,847 [myid:] - INFO [Thread-19:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (1, 0) 2016-07-12 22:35:13,848 [myid:] - INFO [/0.0.0.0:11250:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46883 2016-07-12 22:35:13,849 [myid:] - INFO [/0.0.0.0:11253:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:41748 2016-07-12 22:35:13,849 [myid:] - INFO [Thread-19:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (1, 0) 2016-07-12 22:35:13,849 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,849 [myid:] - INFO [/0.0.0.0:11250:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46885 2016-07-12 22:35:13,849 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,849 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 1, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,850 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,849 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 0, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-12 22:35:13,850 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:13,850 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:13,851 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:13,851 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 12332 2016-07-12 22:35:13,851 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 26 2016-07-12 22:35:13,851 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testCnxManager 2016-07-12 22:35:13,851 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testCnxManager 2016-07-12 22:35:13,851 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxManager 2016-07-12 22:35:13,862 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxManagerTimeout 2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11257 2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11258 2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11259 2016-07-12 22:35:13,862 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11260 2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11261 2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11262 2016-07-12 22:35:13,862 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11263 2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11264 2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11265 2016-07-12 22:35:13,863 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:13,863 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxManagerTimeout 2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11266 2016-07-12 22:35:13,863 [myid:] - INFO [main:CnxManagerTest@171] - This is the dead address I'm trying: 192.0.2.179 2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11267 2016-07-12 22:35:13,863 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 192.0.2.179 to address: /192.0.2.179 2016-07-12 22:35:13,864 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11261 2016-07-12 22:35:13,864 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11262 2016-07-12 22:35:14,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop! 2016-07-12 22:35:14,310 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-12 22:35:14,311 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-12 22:35:14,829 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-12 22:35:14,840 [myid:] - INFO [/0.0.0.0:11244:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-12 22:35:14,842 [myid:] - INFO [/0.0.0.0:11247:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-12 22:35:15,319 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-12 22:35:15,320 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-12 22:35:16,327 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-12 22:35:16,327 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-12 22:35:16,327 [myid:] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-12 22:35:16,328 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-12 22:35:16,328 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-12 22:35:16,328 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-12 22:35:18,869 [myid:] - WARN [main:QuorumCnxManager@400] - Cannot open channel to 2 at election address /192.0.2.179:11267 java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxManagerTimeout(CnxManagerTest.java:187) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-12 22:35:18,870 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 192.0.2.179 to address: /192.0.2.179 2016-07-12 22:35:18,870 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 12750 2016-07-12 22:35:18,870 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 13 2016-07-12 22:35:18,871 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testCnxManagerTimeout 2016-07-12 22:35:18,871 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testCnxManagerTimeout 2016-07-12 22:35:18,871 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxManagerTimeout 2016-07-12 22:35:18,871 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxManagerSpinLock 2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11268 2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11269 2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11270 2016-07-12 22:35:18,872 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11271 2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11272 2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11273 2016-07-12 22:35:18,873 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:18,873 [myid:] - INFO [main:PortAssignment@32] - assigning port 11274 2016-07-12 22:35:18,873 [myid:] - INFO [main:PortAssignment@32] - assigning port 11275 2016-07-12 22:35:18,873 [myid:] - INFO [main:PortAssignment@32] - assigning port 11276 2016-07-12 22:35:18,873 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0 2016-07-12 22:35:18,874 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxManagerSpinLock 2016-07-12 22:35:18,874 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11272 2016-07-12 22:35:18,875 [myid:] - INFO [main:CnxManagerTest@214] - Election port: 11273 2016-07-12 22:35:18,875 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11273 2016-07-12 22:35:19,876 [myid:] - INFO [/0.0.0.0:11273:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:37939 2016-07-12 22:35:19,876 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 1, error = java.io.IOException: Received packet with invalid packet: -20 at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:797) 2016-07-12 22:35:19,877 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-12 22:35:19,877 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-12 22:35:19,877 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-12 22:35:20,876 [myid:] - INFO [main:CnxManagerTest@249] - Socket has been closed as expected 2016-07-12 22:35:20,876 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 12944 2016-07-12 22:35:20,876 [myid:] - ERROR [/0.0.0.0:11273:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-12 22:35:20,877 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 14 2016-07-12 22:35:20,877 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testCnxManagerSpinLock 2016-07-12 22:35:20,877 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testCnxManagerSpinLock 2016-07-12 22:35:20,877 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxManagerSpinLock {noformat}
    via by Michael Han,
    • java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148) at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:267) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:66) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)

    Users with the same issue

    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    andyglickandyglick
    61 times, last one,
    richard77richard77
    2 times, last one,
    23 more bugmates