java.lang.Exception: shutdown Follower

Apache's JIRA Issue Tracker | Michael Han | 5 months ago
  1. 0

    From https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1156 {noformat} Error Message waiting for server up Stacktrace junit.framework.AssertionFailedError: waiting for server up at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:183) at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:113) at org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader(QuorumZxidSyncTest.java:67) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55) Standard Output 2016-07-21 08:11:45,722 [myid:] - INFO [main:PortAssignment@32] - assigning port 11221 2016-07-21 08:11:45,729 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testBehindLeader 2016-07-21 08:11:45,729 [myid:] - INFO [main:QuorumBase@69] - QuorumBase.setup null 2016-07-21 08:11:45,926 [myid:] - INFO [main:PortAssignment@32] - assigning port 11222 2016-07-21 08:11:45,926 [myid:] - INFO [main:PortAssignment@32] - assigning port 11223 2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning port 11224 2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning port 11225 2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning port 11226 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11227 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11228 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11229 2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11230 2016-07-21 08:11:45,929 [myid:] - INFO [main:PortAssignment@32] - assigning port 11231 2016-07-21 08:11:45,929 [myid:] - INFO [main:QuorumBase@93] - Ports are: 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226 2016-07-21 08:11:45,946 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:45,946 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:45,948 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:45,948 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:45,949 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:45,949 [myid:] - INFO [main:QuorumBase@142] - creating QuorumPeer 1 port 11222 2016-07-21 08:11:45,961 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11222 2016-07-21 08:11:45,979 [myid:] - INFO [main:QuorumBase@145] - creating QuorumPeer 2 port 11223 2016-07-21 08:11:45,979 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11223 2016-07-21 08:11:45,980 [myid:] - INFO [main:QuorumBase@148] - creating QuorumPeer 3 port 11224 2016-07-21 08:11:45,980 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11224 2016-07-21 08:11:45,981 [myid:] - INFO [main:QuorumBase@151] - creating QuorumPeer 4 port 11225 2016-07-21 08:11:45,981 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11225 2016-07-21 08:11:45,981 [myid:] - INFO [main:QuorumBase@154] - creating QuorumPeer 5 port 11226 2016-07-21 08:11:45,982 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11226 2016-07-21 08:11:45,982 [myid:] - INFO [main:QuorumBase@163] - QuorumPeer 1 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997} 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@164] - QuorumPeer 2 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997} 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@165] - QuorumPeer 3 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997} 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@166] - QuorumPeer 4 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997} 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@167] - QuorumPeer 5 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997} 2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@169] - start QuorumPeer 1 2016-07-21 08:11:45,988 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:45,990 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:45,996 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12227 2016-07-21 08:11:46,001 [myid:] - INFO [main:QuorumBase@171] - start QuorumPeer 2 2016-07-21 08:11:46,002 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,003 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,004 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12228 2016-07-21 08:11:46,006 [myid:] - INFO [main:QuorumBase@173] - start QuorumPeer 3 2016-07-21 08:11:46,006 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,007 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,007 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@774] - LOOKING 2016-07-21 08:11:46,008 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@774] - LOOKING 2016-07-21 08:11:46,009 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12229 2016-07-21 08:11:46,009 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:FastLeaderElection@818] - New election. My id = 1, proposed zxid=0x0 2016-07-21 08:11:46,009 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:FastLeaderElection@818] - New election. My id = 2, proposed zxid=0x0 2016-07-21 08:11:46,009 [myid:] - INFO [main:QuorumBase@175] - start QuorumPeer 4 2016-07-21 08:11:46,010 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,011 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,011 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@774] - LOOKING 2016-07-21 08:11:46,011 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38924 2016-07-21 08:11:46,011 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@818] - New election. My id = 3, proposed zxid=0x0 2016-07-21 08:11:46,011 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43415 2016-07-21 08:11:46,011 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (2, 1) 2016-07-21 08:11:46,011 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,013 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 1) 2016-07-21 08:11:46,014 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51548 2016-07-21 08:11:46,014 [myid:] - INFO [main:QuorumBase@177] - start QuorumPeer 5 2016-07-21 08:11:46,015 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12230 2016-07-21 08:11:46,015 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,015 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 4 at election address /127.0.0.1:12230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:745) 2016-07-21 08:11:46,018 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,018 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51551 2016-07-21 08:11:46,016 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43417 2016-07-21 08:11:46,016 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@774] - LOOKING 2016-07-21 08:11:46,016 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-21 08:11:46,019 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:FastLeaderElection@818] - New election. My id = 4, proposed zxid=0x0 2016-07-21 08:11:46,019 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46102 2016-07-21 08:11:46,018 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,018 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2) 2016-07-21 08:11:46,018 [myid:] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:46,018 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38928 2016-07-21 08:11:46,021 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2) 2016-07-21 08:11:46,021 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,021 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51553 2016-07-21 08:11:46,020 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,019 [myid:] - INFO [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 3) 2016-07-21 08:11:46,022 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,021 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:745) 2016-07-21 08:11:46,021 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,021 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:745) 2016-07-21 08:11:46,023 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,023 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,023 [myid:] - INFO [WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:46,024 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46104 2016-07-21 08:11:46,022 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12231 2016-07-21 08:11:46,022 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:745) 2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,025 [myid:] - INFO [main:QuorumBase@179] - started QuorumPeer 5 2016-07-21 08:11:46,026 [myid:] - INFO [main:QuorumBase@181] - Checking ports 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226 2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,025 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43428 2016-07-21 08:11:46,027 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46109 2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,024 [myid:] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:46,027 [myid:] - INFO [WorkerSender[myid=4]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 4) 2016-07-21 08:11:46,027 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,026 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38938 2016-07-21 08:11:46,025 [myid:] - INFO [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1 2016-07-21 08:11:46,025 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2) 2016-07-21 08:11:46,028 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11222 2016-07-21 08:11:46,028 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 4, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:11:46,029 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:11:46,028 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,027 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@774] - LOOKING 2016-07-21 08:11:46,027 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,027 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52086 2016-07-21 08:11:46,030 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 2, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:11:46,030 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@732] - Exception when using channel: for id 4 my id = 2 error = java.net.SocketException: Broken pipe 2016-07-21 08:11:46,029 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,031 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52363 2016-07-21 08:11:46,029 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38940 2016-07-21 08:11:46,029 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1) 2016-07-21 08:11:46,029 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,032 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,032 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,029 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@732] - Exception when using channel: for id 2 my id = 4 error = java.net.SocketException: Socket closed 2016-07-21 08:11:46,033 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:11:46,029 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2) 2016-07-21 08:11:46,028 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,033 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1) 2016-07-21 08:11:46,032 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,031 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52088 2016-07-21 08:11:46,031 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,031 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46117 2016-07-21 08:11:46,030 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:11:46,030 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:11:46,030 [myid:] - INFO [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 3) 2016-07-21 08:11:46,030 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:FastLeaderElection@818] - New election. My id = 5, proposed zxid=0x0 2016-07-21 08:11:46,030 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,034 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38947 2016-07-21 08:11:46,034 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,034 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,033 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,035 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52089 2016-07-21 08:11:46,035 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,037 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43442 2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,038 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52091 2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,036 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1) 2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,038 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51573 2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,039 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52093 2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,041 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43444 2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,041 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.DataInputStream.readFully(DataInputStream.java:195) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:805) 2016-07-21 08:11:46,042 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,041 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:11:46,041 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxn@827] - Processing stat command from /127.0.0.1:52363 2016-07-21 08:11:46,041 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 1, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:11:46,043 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,043 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,043 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:11:46,042 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,042 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:11:46,044 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:11:46,042 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,042 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52096 2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,045 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:11:46,045 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:11:46,045 [myid:] - INFO [Thread-2:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52363 (no session established for client) 2016-07-21 08:11:46,044 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,044 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,044 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,046 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,045 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43446 2016-07-21 08:11:46,045 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,045 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 1, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:11:46,047 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:11:46,045 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:11:46,047 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:11:46,047 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,047 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:11:46,048 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:11:46,046 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,048 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,048 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,049 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,049 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,049 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state) 2016-07-21 08:11:46,245 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@844] - FOLLOWING 2016-07-21 08:11:46,246 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@844] - FOLLOWING 2016-07-21 08:11:46,247 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@844] - FOLLOWING 2016-07-21 08:11:46,249 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@844] - FOLLOWING 2016-07-21 08:11:46,249 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@856] - LEADING 2016-07-21 08:11:46,251 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Learner@86] - TCP NoDelay set to: true 2016-07-21 08:11:46,255 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:Leader@59] - TCP NoDelay set to: true 2016-07-21 08:11:46,259 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016 07:46 GMT 2016-07-21 08:11:46,259 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:host.name=asf907.gq1.ygridcore.net 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.version=1.7.0_80 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.vendor=Oracle Corporation 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.io.tmpdir=/tmp 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.compiler=<NA> 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.name=Linux 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.arch=amd64 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.version=3.13.0-36-lowlatency 2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0: ...[truncated 1420555 chars]... ager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:21,678 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:21,672 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:21,679 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:21,672 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:21,678 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:21,677 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:21,680 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:21,675 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:21,680 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop! 2016-07-21 08:14:22,317 [myid:] - INFO [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-21 08:14:22,320 [myid:] - INFO [/127.0.0.1:12242:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-21 08:14:22,579 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-21 08:14:22,579 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:22,579 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:22,579 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-21 08:14:22,582 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 2016-07-21 08:14:22,582 [myid:] - INFO [main:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307) at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:298) at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285) at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030) 2016-07-21 08:14:22,582 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:22,582 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-21 08:14:22,582 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down 2016-07-21 08:14:22,582 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-21 08:14:22,582 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop! 2016-07-21 08:14:22,582 [myid:] - INFO [CommitProcessor:2:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-21 08:14:22,582 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-21 08:14:22,583 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:22,583 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-21 08:14:22,584 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11234:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 4, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 2, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:22,586 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 5, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:22,586 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 2, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:22,587 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 3, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:22,587 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:22,587 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,585 [myid:] - ERROR [/127.0.0.1:12239:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-21 08:14:22,587 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:22,588 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,587 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:22,589 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,586 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:22,589 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,586 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 2, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:22,589 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:22,586 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 2016-07-21 08:14:22,589 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 to exit thread 2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:22,590 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:22,673 [myid:] - INFO [/127.0.0.1:12238:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-21 08:14:23,079 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-21 08:14:23,080 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-21 08:14:23,084 [myid:] - INFO [WorkerSender[myid=5]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-21 08:14:23,088 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-21 08:14:23,088 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-21 08:14:23,089 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-21 08:14:23,578 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-21 08:14:23,578 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:23,578 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:23,578 [myid:] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-21 08:14:23,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 2016-07-21 08:14:23,579 [myid:] - INFO [main:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307) at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:299) at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285) at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030) 2016-07-21 08:14:23,579 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:23,579 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-21 08:14:23,580 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down 2016-07-21 08:14:23,580 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-21 08:14:23,580 [myid:] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop! 2016-07-21 08:14:23,580 [myid:] - INFO [CommitProcessor:3:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-21 08:14:23,580 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-21 08:14:23,581 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:23,581 [myid:] - INFO [SyncThread:3:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-21 08:14:23,582 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11235:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-21 08:14:23,583 [myid:] - ERROR [/127.0.0.1:12240:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-21 08:14:23,583 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:23,585 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 3, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:23,585 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:23,584 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:23,585 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 3, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:23,585 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 5, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:23,586 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 4, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:23,586 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:23,584 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 2016-07-21 08:14:23,586 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 to exit thread 2016-07-21 08:14:23,586 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:23,586 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:23,586 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:23,587 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:23,588 [myid:] - INFO [/127.0.0.1:12239:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-21 08:14:24,355 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-21 08:14:24,358 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-21 08:14:24,360 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down 2016-07-21 08:14:24,361 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-21 08:14:24,361 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-21 08:14:24,362 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down 2016-07-21 08:14:24,578 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-21 08:14:24,578 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:24,578 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:24,578 [myid:] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-21 08:14:24,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 2016-07-21 08:14:24,579 [myid:] - INFO [main:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307) at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:300) at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285) at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030) 2016-07-21 08:14:24,580 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:24,581 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-21 08:14:24,581 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down 2016-07-21 08:14:24,581 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-21 08:14:24,581 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-21 08:14:24,581 [myid:] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop! 2016-07-21 08:14:24,581 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:24,581 [myid:] - INFO [CommitProcessor:4:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-21 08:14:24,582 [myid:] - INFO [SyncThread:4:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-21 08:14:24,583 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-21 08:14:24,585 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 2016-07-21 08:14:24,586 [myid:] - ERROR [/127.0.0.1:12241:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-21 08:14:24,586 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:24,586 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:24,586 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 to exit thread 2016-07-21 08:14:24,586 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 5, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:24,587 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:24,585 [myid:] - INFO [/127.0.0.1:12240:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-21 08:14:24,585 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 4, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795) 2016-07-21 08:14:24,588 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker 2016-07-21 08:14:24,588 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715) 2016-07-21 08:14:24,588 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread 2016-07-21 08:14:25,579 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850) 2016-07-21 08:14:25,579 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:FollowerZooKeeperServer@140] - Shutting down 2016-07-21 08:14:25,579 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:25,579 [myid:] - WARN [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-21 08:14:25,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 2016-07-21 08:14:25,579 [myid:] - INFO [main:Leader@496] - Shutting down 2016-07-21 08:14:25,580 [myid:] - INFO [main:Leader@502] - Shutdown called java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307) at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:301) at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285) at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030) 2016-07-21 08:14:25,581 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down 2016-07-21 08:14:25,581 [myid:] - INFO [LearnerCnxAcceptor-/127.0.0.1:12237:Leader$LearnerCnxAcceptor@325] - exception while shutting down acceptor: java.net.SocketException: Socket closed 2016-07-21 08:14:25,582 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down 2016-07-21 08:14:25,582 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down 2016-07-21 08:14:25,582 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down 2016-07-21 08:14:25,582 [myid:] - INFO [main:CommitProcessor@184] - Shutting down 2016-07-21 08:14:25,582 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! 2016-07-21 08:14:25,583 [myid:] - INFO [CommitProcessor:5:CommitProcessor@153] - CommitProcessor exited loop! 2016-07-21 08:14:25,582 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down 2016-07-21 08:14:25,583 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete 2016-07-21 08:14:25,583 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down 2016-07-21 08:14:25,583 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@187] - SyncRequestProcessor exited! 2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43485:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43485 ******** 2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43486:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43486 ******** 2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43486:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43485:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43488:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43488 ******** 2016-07-21 08:14:25,586 [myid:] - INFO [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener 2016-07-21 08:14:25,586 [myid:] - WARN [LearnerHandler-/127.0.0.1:43487:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43487 ******** 2016-07-21 08:14:25,586 [myid:] - WARN [LearnerHandler-/127.0.0.1:43487:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-21 08:14:25,586 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method 2016-07-21 08:14:25,586 [myid:] - WARN [LearnerHandler-/127.0.0.1:43488:LearnerHandler@656] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647) 2016-07-21 08:14:25,587 [myid:] - WARN [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@862] - Unexpected exception java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859) 2016-07-21 08:14:25,588 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:Leader@496] - Shutting down 2016-07-21 08:14:25,588 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 2016-07-21 08:14:25,588 [myid:] - ERROR [/127.0.0.1:12242:QuorumCnxManager$Listener@547] - Exception while listening java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539) 2016-07-21 08:14:25,588 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 to exit thread 2016-07-21 08:14:25,588 [myid:] - WARN [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@874] - QuorumPeer main thread exited 2016-07-21 08:14:25,589 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11233 2016-07-21 08:14:25,589 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11233 is no longer accepting client connections 2016-07-21 08:14:25,589 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11234 2016-07-21 08:14:25,589 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11234 is no longer accepting client connections 2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11235 2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11235 is no longer accepting client connections 2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11236 2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11236 is no longer accepting client connections 2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11237 2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11237 is no longer accepting client connections 2016-07-21 08:14:25,592 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testLateLogs 2016-07-21 08:14:25,592 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testLateLogs {noformat}

    Apache's JIRA Issue Tracker | 5 months ago | Michael Han
    java.lang.Exception: shutdown Follower
  2. 0

    Two tests related to ReconfigTest fail consistently on Solaris: org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortLeaderorg.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortFollower Examples: https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-solaris/1245/#showFailuresLink https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_solaris/185/#showFailuresLink {noformat} org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortFollower Failing for the past 5 builds (Since Failed#1241 ) Took 1 min 37 sec. Error Message client could not connect to reestablished quorum: giving up after 30+ seconds. Stacktrace junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds. at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:173) at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPort(ReconfigTest.java:732) at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortFollower(ReconfigTest.java:658) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) Standard Output 2016-07-28 09:24:37,121 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods. 2016-07-28 09:24:37,185 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods. 2016-07-28 09:24:37,200 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testQuorumSystemChange 2016-07-28 09:24:37,202 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testQuorumSystemChange 2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@157] - Single test process using ports from 11221 - 32767. 2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11222 from range 11221 - 32767. 2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11223 from range 11221 - 32767. 2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11224 from range 11221 - 32767. 2016-07-28 09:24:37,580 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11225 from range 11221 - 32767. 2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11226 from range 11221 - 32767. 2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11227 from range 11221 - 32767. 2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11228 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11229 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11230 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11231 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11232 from range 11221 - 32767. 2016-07-28 09:24:37,584 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11233 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11234 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11235 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11236 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11237 from range 11221 - 32767. 2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11238 from range 11221 - 32767. 2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11239 from range 11221 - 32767. 2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11240 from range 11221 - 32767. 2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11241 from range 11221 - 32767. 2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11242 from range 11221 - 32767. 2016-07-28 09:24:37,587 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 1; public port 11222 2016-07-28 09:24:37,609 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,615 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222 2016-07-28 09:24:37,638 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 2; public port 11225 2016-07-28 09:24:37,638 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,639 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225 2016-07-28 09:24:37,639 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 3; public port 11228 2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228 2016-07-28 09:24:37,641 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 4; public port 11231 2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231 2016-07-28 09:24:37,642 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 5; public port 11234 2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11234 2016-07-28 09:24:37,643 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 6; public port 11237 2016-07-28 09:24:37,643 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11237 2016-07-28 09:24:37,644 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 7; public port 11240 2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,645 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11240 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 2016-07-28 09:24:37,651 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11222 is no longer accepting client connections 2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11225 2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11225 is no longer accepting client connections 2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11228 2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11228 is no longer accepting client connections 2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11231 2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11231 is no longer accepting client connections 2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11234 2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11234 is no longer accepting client connections 2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11237 2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11237 is no longer accepting client connections 2016-07-28 09:24:37,654 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11240 2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11240 is no longer accepting client connections 2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 1; public port 11222 2016-07-28 09:24:37,654 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,655 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222 2016-07-28 09:24:37,660 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,708 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 1 2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 2; public port 11225 2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225 2016-07-28 09:24:37,756 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,759 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11224 2016-07-28 09:24:37,771 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 2 2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 3; public port 11228 2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228 2016-07-28 09:24:37,781 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11227 2016-07-28 09:24:37,888 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0 2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0 2016-07-28 09:24:37,899 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50183 2016-07-28 09:24:37,899 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1) 2016-07-28 09:24:37,901 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50185 2016-07-28 09:24:37,901 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,905 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,905 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,907 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,908 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,920 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,961 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,908 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,965 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,967 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,968 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,968 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,981 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,982 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,015 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 3 2016-07-28 09:24:38,016 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 4; public port 11231 2016-07-28 09:24:38,016 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:38,017 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231 2016-07-28 09:24:38,017 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-07-28 09:24:38,018 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x0 2016-07-28 09:24:38,018 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:38,020 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11230 2016-07-28 09:24:38,025 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50200 2016-07-28 09:24:38,026 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50201 2016-07-28 09:24:38,027 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,027 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,028 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,033 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,032 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,034 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,035 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,036 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,037 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,037 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,038 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,044 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ...[truncated 3451538 chars]... mPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.2] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.3] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.5] 2016-07-28 09:30:21,332 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) 2016-07-28 09:30:21,332 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:21,333 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-07-28 09:30:21,333 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower] 2016-07-28 09:30:21,333 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-07-28 09:30:21,333 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-07-28 09:30:21,333 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-07-28 09:30:21,333 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:21,334 [myid:] - INFO [CommitProcessor:2:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:21,334 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:21,334 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree] 2016-07-28 09:30:21,334 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:21,334 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:21,334 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:21,335 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2] 2016-07-28 09:30:21,335 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2] 2016-07-28 09:30:21,336 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11374:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.1] 2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.3] 2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.5] 2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,337 [myid:] - INFO [/127.0.0.1:11376:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:21,338 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) 2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) to exit thread 2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) 2016-07-28 09:30:21,339 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:21,339 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-07-28 09:30:21,339 [myid:] - INFO [CommitProcessor:3:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:21,339 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:21,340 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=InMemoryDataTree] 2016-07-28 09:30:21,340 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:21,340 [myid:] - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:21,340 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11377:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:21,342 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,342 [myid:] - INFO [/127.0.0.1:11379:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) 2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) to exit thread 2016-07-28 09:30:21,421 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273 2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257 2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276 2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - INFO [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11306. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - ERROR [main-SendThread(127.0.0.1:11306):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11306 2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - WARN [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1235] - Session 0x222add109d00000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263 2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303 2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - INFO [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11270. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - ERROR [main-SendThread(127.0.0.1:11270):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11270 2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - WARN [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1235] - Session 0x122add08ccf0000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower] 2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.1] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.2] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.5] 2016-07-28 09:30:22,342 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) 2016-07-28 09:30:22,342 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:22,342 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-07-28 09:30:22,342 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:22,343 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-07-28 09:30:22,343 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:22,343 [myid:] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-07-28 09:30:22,343 [myid:] - INFO [CommitProcessor:4:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:22,343 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:22,344 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower,name3=InMemoryDataTree] 2016-07-28 09:30:22,344 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:22,344 [myid:] - INFO [SyncThread:4:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:22,344 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11380:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:22,346 [myid:] - INFO [/127.0.0.1:11382:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) 2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) to exit thread 2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - INFO [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11309. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - ERROR [main-SendThread(127.0.0.1:11309):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11309 2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - WARN [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1235] - Session 0x322add109d50000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273 2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,761 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,761 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - INFO [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11260. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - ERROR [main-SendThread(127.0.0.1:11260):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11260 2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - WARN [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1235] - Session 0x222adcf0da20000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263 2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303 2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276 2016-07-28 09:30:23,232 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257 2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower] 2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.1] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.2] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.3] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.5] 2016-07-28 09:30:23,352 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) 2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@617] - Shutting down 2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@623] - Shutdown called java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:623) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1181) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:23,353 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [main:LeaderRequestProcessor@77] - Shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [LearnerCnxAcceptor-/127.0.0.1:11384:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.SocketException: Socket closed 2016-07-28 09:30:23,353 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop! 2016-07-28 09:30:23,353 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [CommitProcessor:5:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:23,354 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@918] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:23,354 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:23,355 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader,name3=InMemoryDataTree] 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51992 ******** 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51977 ******** 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51976 ******** 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51975 ******** 2016-07-28 09:30:23,356 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11383:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,357 [myid:] - INFO [/127.0.0.1:11385:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:23,357 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader] 2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) 2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1127] - Unexpected exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:561) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1124) 2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):Leader@617] - Shutting down 2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) to exit thread 2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.1] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.2] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.3] 2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11371 2016-07-28 09:30:23,359 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11371 is no longer accepting client connections 2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11374 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11374 is no longer accepting client connections 2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11377 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11377 is no longer accepting client connections 2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11380 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11380 is no longer accepting client connections 2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11383 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11383 is no longer accepting client connections 2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testRemoveOneAsynchronous 2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testRemoveOneAsynchronous {noformat} {noformat} Failed org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortLeader Failing for the past 5 builds (Since Failed#1241 ) Took 1 min 38 sec. Error Message client could not connect to reestablished quorum: giving up after 30+ seconds. Stacktrace junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds. at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:173) at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPort(ReconfigTest.java:732) at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortLeader(ReconfigTest.java:662) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) Standard Output 2016-07-28 09:24:37,121 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods. 2016-07-28 09:24:37,185 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods. 2016-07-28 09:24:37,200 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testQuorumSystemChange 2016-07-28 09:24:37,202 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testQuorumSystemChange 2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@157] - Single test process using ports from 11221 - 32767. 2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11222 from range 11221 - 32767. 2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11223 from range 11221 - 32767. 2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11224 from range 11221 - 32767. 2016-07-28 09:24:37,580 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11225 from range 11221 - 32767. 2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11226 from range 11221 - 32767. 2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11227 from range 11221 - 32767. 2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11228 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11229 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11230 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11231 from range 11221 - 32767. 2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11232 from range 11221 - 32767. 2016-07-28 09:24:37,584 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11233 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11234 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11235 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11236 from range 11221 - 32767. 2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11237 from range 11221 - 32767. 2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11238 from range 11221 - 32767. 2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11239 from range 11221 - 32767. 2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11240 from range 11221 - 32767. 2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11241 from range 11221 - 32767. 2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11242 from range 11221 - 32767. 2016-07-28 09:24:37,587 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 1; public port 11222 2016-07-28 09:24:37,609 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,615 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222 2016-07-28 09:24:37,638 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 2; public port 11225 2016-07-28 09:24:37,638 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,639 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225 2016-07-28 09:24:37,639 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 3; public port 11228 2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228 2016-07-28 09:24:37,641 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 4; public port 11231 2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231 2016-07-28 09:24:37,642 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 5; public port 11234 2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11234 2016-07-28 09:24:37,643 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 6; public port 11237 2016-07-28 09:24:37,643 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11237 2016-07-28 09:24:37,644 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 7; public port 11240 2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,645 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11240 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer 2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer 2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread 2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 2016-07-28 09:24:37,651 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11222 is no longer accepting client connections 2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11225 2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11225 is no longer accepting client connections 2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11228 2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11228 is no longer accepting client connections 2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11231 2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11231 is no longer accepting client connections 2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11234 2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11234 is no longer accepting client connections 2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11237 2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11237 is no longer accepting client connections 2016-07-28 09:24:37,654 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11240 2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11240 is no longer accepting client connections 2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 1; public port 11222 2016-07-28 09:24:37,654 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,655 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222 2016-07-28 09:24:37,660 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,708 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 1 2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 2; public port 11225 2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225 2016-07-28 09:24:37,756 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,759 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11224 2016-07-28 09:24:37,771 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 2 2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 3; public port 11228 2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228 2016-07-28 09:24:37,781 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11227 2016-07-28 09:24:37,888 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0 2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0 2016-07-28 09:24:37,899 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50183 2016-07-28 09:24:37,899 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1) 2016-07-28 09:24:37,901 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50185 2016-07-28 09:24:37,901 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,905 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,905 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,907 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,908 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,920 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:37,961 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,908 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,965 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,967 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,968 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:37,968 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,981 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:37,982 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,015 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 3 2016-07-28 09:24:38,016 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 4; public port 11231 2016-07-28 09:24:38,016 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2016-07-28 09:24:38,017 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231 2016-07-28 09:24:38,017 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-07-28 09:24:38,018 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x0 2016-07-28 09:24:38,018 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-07-28 09:24:38,020 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11230 2016-07-28 09:24:38,025 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50200 2016-07-28 09:24:38,026 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50201 2016-07-28 09:24:38,027 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,027 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,028 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,033 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,032 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,034 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,035 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,036 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,037 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-07-28 09:24:38,037 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,038 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,044 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ...[truncated 3451538 chars]... mPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.2] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.3] 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.5] 2016-07-28 09:30:21,332 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) 2016-07-28 09:30:21,332 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:21,333 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-07-28 09:30:21,333 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower] 2016-07-28 09:30:21,333 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-07-28 09:30:21,333 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-07-28 09:30:21,333 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-07-28 09:30:21,333 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:21,334 [myid:] - INFO [CommitProcessor:2:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:21,334 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:21,334 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree] 2016-07-28 09:30:21,334 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:21,334 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:21,334 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:21,335 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2] 2016-07-28 09:30:21,335 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2] 2016-07-28 09:30:21,336 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11374:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.1] 2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.3] 2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.5] 2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,337 [myid:] - INFO [/127.0.0.1:11376:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:21,338 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) 2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) to exit thread 2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) 2016-07-28 09:30:21,339 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:21,339 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:21,339 [myid:] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-07-28 09:30:21,339 [myid:] - INFO [CommitProcessor:3:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:21,339 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:21,340 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=InMemoryDataTree] 2016-07-28 09:30:21,340 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:21,340 [myid:] - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:21,340 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11377:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:21,342 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:21,342 [myid:] - INFO [/127.0.0.1:11379:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) 2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) to exit thread 2016-07-28 09:30:21,421 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273 2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257 2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276 2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - INFO [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11306. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - ERROR [main-SendThread(127.0.0.1:11306):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11306 2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - WARN [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1235] - Session 0x222add109d00000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263 2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303 2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - INFO [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11270. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - ERROR [main-SendThread(127.0.0.1:11270):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11270 2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - WARN [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1235] - Session 0x122add08ccf0000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower] 2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.1] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.2] 2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.5] 2016-07-28 09:30:22,342 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) 2016-07-28 09:30:22,342 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:22,342 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-07-28 09:30:22,342 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:22,343 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-07-28 09:30:22,343 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:22,343 [myid:] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-07-28 09:30:22,343 [myid:] - INFO [CommitProcessor:4:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:22,343 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:22,344 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower,name3=InMemoryDataTree] 2016-07-28 09:30:22,344 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:22,344 [myid:] - INFO [SyncThread:4:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:22,344 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11380:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:22,346 [myid:] - INFO [/127.0.0.1:11382:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) 2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) to exit thread 2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - INFO [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11309. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - ERROR [main-SendThread(127.0.0.1:11309):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11309 2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - WARN [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1235] - Session 0x322add109d50000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273 2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:22,761 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,761 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - INFO [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11260. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - ERROR [main-SendThread(127.0.0.1:11260):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11260 2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - WARN [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1235] - Session 0x222adcf0da20000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263 2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303 2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276 2016-07-28 09:30:23,232 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error) 2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257 2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133) 2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower] 2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.1] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.2] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.3] 2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.5] 2016-07-28 09:30:23,352 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) 2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@617] - Shutting down 2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@623] - Shutdown called java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:623) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1181) at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251) at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238) at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306) at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) 2016-07-28 09:30:23,353 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [main:LeaderRequestProcessor@77] - Shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down 2016-07-28 09:30:23,353 [myid:] - INFO [LearnerCnxAcceptor-/127.0.0.1:11384:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.SocketException: Socket closed 2016-07-28 09:30:23,353 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop! 2016-07-28 09:30:23,353 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [CommitProcessor:5:CommitProcessor@299] - CommitProcessor exited loop! 2016-07-28 09:30:23,354 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@918] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-07-28 09:30:23,354 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-07-28 09:30:23,354 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-07-28 09:30:23,355 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader,name3=InMemoryDataTree] 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51992 ******** 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51977 ******** 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51976 ******** 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51975 ******** 2016-07-28 09:30:23,356 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11383:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-07-28 09:30:23,357 [myid:] - INFO [/127.0.0.1:11385:QuorumCnxManager$Listener@661] - Leaving listener 2016-07-28 09:30:23,357 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader] 2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) 2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1127] - Unexpected exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:561) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1124) 2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):Leader@617] - Shutting down 2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) to exit thread 2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.1] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.2] 2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.3] 2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11371 2016-07-28 09:30:23,359 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11371 is no longer accepting client connections 2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11374 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11374 is no longer accepting client connections 2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11377 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11377 is no longer accepting client connections 2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11380 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11380 is no longer accepting client connections 2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11383 2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11383 is no longer accepting client connections 2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testRemoveOneAsynchronous 2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testRemoveOneAsynchronous {noformat}

    Apache's JIRA Issue Tracker | 4 months ago | Michael Han
    java.lang.Exception: shutdown Follower
  3. 0

    This failure is new and consistent on jdk7/8 with trunk branch - happened after build 3070 recently. Not sure if this is caused by svn - git migration or not. {noformat} Error Message zoo.cfg.dynamic.next is not deleted. Stacktrace junit.framework.AssertionFailedError: zoo.cfg.dynamic.next is not deleted. at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:155) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) Standard Output 2016-09-13 05:09:25,247 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods. 2016-09-13 05:09:25,349 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods. 2016-09-13 05:09:25,370 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testDuringLeaderSync 2016-09-13 05:09:25,372 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testDuringLeaderSync 2016-09-13 05:09:25,375 [myid:] - INFO [main:PortAssignment@151] - Test process 2/8 using ports from 13914 - 16606. 2016-09-13 05:09:25,380 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13915 from range 13914 - 16606. 2016-09-13 05:09:25,380 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13916 from range 13914 - 16606. 2016-09-13 05:09:25,381 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13917 from range 13914 - 16606. 2016-09-13 05:09:25,381 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13918 from range 13914 - 16606. 2016-09-13 05:09:25,381 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13919 from range 13914 - 16606. 2016-09-13 05:09:25,382 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13920 from range 13914 - 16606. 2016-09-13 05:09:25,382 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13921 from range 13914 - 16606. 2016-09-13 05:09:25,382 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13922 from range 13914 - 16606. 2016-09-13 05:09:25,383 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13923 from range 13914 - 16606. 2016-09-13 05:09:25,406 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 0 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir clientPort = 13915 adminServerPort = 8080 2016-09-13 05:09:25,416 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 1 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir clientPort = 13918 adminServerPort = 8080 2016-09-13 05:09:25,420 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 2 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir clientPort = 13921 adminServerPort = 8080 2016-09-13 05:09:25,422 [myid:] - INFO [Thread-0:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/zoo.cfg 2016-09-13 05:09:25,422 [myid:] - INFO [Thread-2:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/zoo.cfg 2016-09-13 05:09:25,422 [myid:] - INFO [Thread-1:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/zoo.cfg 2016-09-13 05:09:25,424 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915 2016-09-13 05:09:25,425 [myid:] - INFO [Thread-0:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:13915 2016-09-13 05:09:25,425 [myid:] - INFO [Thread-0:QuorumPeerConfig@322] - secureClientPort is not set 2016-09-13 05:09:25,425 [myid:] - INFO [Thread-1:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:13918 2016-09-13 05:09:25,425 [myid:] - INFO [Thread-1:QuorumPeerConfig@322] - secureClientPort is not set 2016-09-13 05:09:25,425 [myid:] - INFO [Thread-2:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:13921 2016-09-13 05:09:25,426 [myid:] - INFO [Thread-2:QuorumPeerConfig@322] - secureClientPort is not set 2016-09-13 05:09:25,430 [myid:] - INFO [main:ClientBase@248] - server 127.0.0.1:13915 not up java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99) at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69) at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241) at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232) at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) 2016-09-13 05:09:25,444 [myid:1] - INFO [Thread-1:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2016-09-13 05:09:25,445 [myid:2] - INFO [Thread-2:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2016-09-13 05:09:25,445 [myid:2] - INFO [Thread-2:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2016-09-13 05:09:25,444 [myid:0] - INFO [Thread-0:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2016-09-13 05:09:25,445 [myid:2] - INFO [Thread-2:DatadirCleanupManager@101] - Purge task is not scheduled. 2016-09-13 05:09:25,445 [myid:1] - INFO [Thread-1:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2016-09-13 05:09:25,446 [myid:1] - INFO [Thread-1:DatadirCleanupManager@101] - Purge task is not scheduled. 2016-09-13 05:09:25,445 [myid:0] - INFO [Thread-0:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2016-09-13 05:09:25,446 [myid:0] - INFO [Thread-0:DatadirCleanupManager@101] - Purge task is not scheduled. 2016-09-13 05:09:25,446 [myid:2] - INFO [Thread-2:ManagedUtil@46] - Log4j found with jmx enabled. 2016-09-13 05:09:25,447 [myid:0] - INFO [Thread-0:ManagedUtil@46] - Log4j found with jmx enabled. 2016-09-13 05:09:25,446 [myid:1] - INFO [Thread-1:ManagedUtil@46] - Log4j found with jmx enabled. 2016-09-13 05:09:25,552 [myid:1] - ERROR [Thread-1:ManagedUtil@114] - Problems while registering log4j jmx beans! javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:25,552 [myid:1] - WARN [Thread-1:QuorumPeerMain@133] - Unable to register log4j JMX control javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:25,553 [myid:1] - INFO [Thread-1:QuorumPeerMain@136] - Starting quorum peer 2016-09-13 05:09:25,553 [myid:2] - ERROR [Thread-2:HierarchyDynamicMBean@138] - Could not add loggerMBean for [root]. javax.management.InstanceAlreadyExistsException: log4j:logger=root at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160) at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125) at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:25,559 [myid:0] - ERROR [Thread-0:ManagedUtil@114] - Problems while registering log4j jmx beans! javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:25,560 [myid:0] - WARN [Thread-0:QuorumPeerMain@133] - Unable to register log4j JMX control javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:25,560 [myid:0] - INFO [Thread-0:QuorumPeerMain@136] - Starting quorum peer 2016-09-13 05:09:25,562 [myid:2] - INFO [Thread-2:QuorumPeerMain@136] - Starting quorum peer 2016-09-13 05:09:25,581 [myid:0] - INFO [Thread-0:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. 2016-09-13 05:09:25,581 [myid:2] - INFO [Thread-2:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. 2016-09-13 05:09:25,585 [myid:1] - INFO [Thread-1:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. 2016-09-13 05:09:25,591 [myid:0] - INFO [Thread-0:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:13915 2016-09-13 05:09:25,592 [myid:1] - INFO [Thread-1:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:13918 2016-09-13 05:09:25,593 [myid:2] - INFO [Thread-2:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:13921 2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1327] - Local sessions disabled 2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1338] - Local session upgrading disabled 2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1305] - tickTime set to 4000 2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1349] - minSessionTimeout set to 8000 2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1360] - maxSessionTimeout set to 80000 2016-09-13 05:09:25,622 [myid:1] - INFO [Thread-1:QuorumPeer@1327] - Local sessions disabled 2016-09-13 05:09:25,622 [myid:1] - INFO [Thread-1:QuorumPeer@1338] - Local session upgrading disabled 2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1305] - tickTime set to 4000 2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1349] - minSessionTimeout set to 8000 2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1360] - maxSessionTimeout set to 80000 2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1375] - initLimit set to 10 2016-09-13 05:09:25,622 [myid:0] - INFO [Thread-0:QuorumPeer@1327] - Local sessions disabled 2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1338] - Local session upgrading disabled 2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1305] - tickTime set to 4000 2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1349] - minSessionTimeout set to 8000 2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1360] - maxSessionTimeout set to 80000 2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1375] - initLimit set to 10 2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1375] - initLimit set to 10 2016-09-13 05:09:25,666 [myid:0] - INFO [Thread-0:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-09-13 05:09:25,668 [myid:1] - INFO [Thread-1:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-09-13 05:09:25,668 [myid:2] - INFO [Thread-2:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-09-13 05:09:25,669 [myid:0] - INFO [Thread-0:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-09-13 05:09:25,672 [myid:1] - INFO [Thread-1:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-09-13 05:09:25,677 [myid:2] - INFO [Thread-2:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation 2016-09-13 05:09:25,689 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915 2016-09-13 05:09:25,704 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38366 2016-09-13 05:09:25,758 [myid:0] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:38366 2016-09-13 05:09:25,770 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13920 2016-09-13 05:09:25,772 [myid:0] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38366 (no session established for client) 2016-09-13 05:09:25,783 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-09-13 05:09:25,785 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0 2016-09-13 05:09:25,797 [myid:0] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13917 2016-09-13 05:09:25,798 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-09-13 05:09:25,798 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):FastLeaderElection@894] - New election. My id = 0, proposed zxid=0x0 2016-09-13 05:09:25,799 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53023 2016-09-13 05:09:25,806 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-09-13 05:09:25,806 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13923 2016-09-13 05:09:25,806 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0 2016-09-13 05:09:25,825 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:58679 2016-09-13 05:09:25,826 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0) 2016-09-13 05:09:25,825 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1) 2016-09-13 05:09:25,827 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53026 2016-09-13 05:09:25,827 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,827 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,848 [myid:1] - INFO [/127.0.0.1:13920:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:46423 2016-09-13 05:09:25,848 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,857 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:58680 2016-09-13 05:09:25,857 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,858 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 2, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900) 2016-09-13 05:09:25,858 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:25,862 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,863 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,865 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 0 error = java.net.SocketException: Broken pipe 2016-09-13 05:09:25,867 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:25,898 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820) 2016-09-13 05:09:25,898 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,898 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,899 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0 2016-09-13 05:09:25,899 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,900 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,901 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,901 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53028 2016-09-13 05:09:25,898 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2 2016-09-13 05:09:25,901 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,902 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0) 2016-09-13 05:09:25,903 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,903 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,904 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:58684 2016-09-13 05:09:25,905 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820) 2016-09-13 05:09:25,905 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2 2016-09-13 05:09:25,905 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:25,907 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 0 error = java.net.SocketException: Broken pipe 2016-09-13 05:09:25,907 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53030 2016-09-13 05:09:25,908 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0 2016-09-13 05:09:25,908 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:25,909 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:25,910 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2016-09-13 05:09:26,022 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915 2016-09-13 05:09:26,023 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38382 2016-09-13 05:09:26,032 [myid:0] - INFO [NIOWorkerThread-2:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:38382 2016-09-13 05:09:26,033 [myid:0] - INFO [NIOWorkerThread-2:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38382 (no session established for client) 2016-09-13 05:09:26,103 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection] 2016-09-13 05:09:26,104 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1109] - FOLLOWING 2016-09-13 05:09:26,109 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=LeaderElection] 2016-09-13 05:09:26,110 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@88] - TCP NoDelay set to: true 2016-09-13 05:09:26,110 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1121] - LEADING 2016-09-13 05:09:26,111 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=LeaderElection] 2016-09-13 05:09:26,111 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1109] - FOLLOWING 2016-09-13 05:09:26,113 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@63] - TCP NoDelay set to: true 2016-09-13 05:09:26,113 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@83] - zookeeper.leader.maxConcurrentSnapshots = 10 2016-09-13 05:09:26,113 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@85] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5 2016-09-13 05:09:26,122 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.6.0-SNAPSHOT--1, built on 09/13/2016 05:08 GMT 2016-09-13 05:09:26,122 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:host.name=jenkins-test-5f3 2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.version=1.7.0_101 2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation 2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre 2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar 2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmp 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA> 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.name=Linux 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.arch=amd64 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.version=3.13.0-30-generic 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:user.name=jenkins 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:user.home=/home/jenkins 2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test 2016-09-13 05:09:26,125 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.memory.free=49MB 2016-09-13 05:09:26,125 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.memory.max=455MB 2016-09-13 05:09:26,125 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.memory.total=60MB 2016-09-13 05:09:26,126 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000 2016-09-13 05:09:26,127 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000 2016-09-13 05:09:26,127 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/data/version-2 2016-09-13 05:09:26,126 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000 2016-09-13 05:09:26,129 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000 2016-09-13 05:09:26,129 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/data/version-2 2016-09-13 05:09:26,129 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 18 MS 2016-09-13 05:09:26,126 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000 2016-09-13 05:09:26,129 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@412] - LEADING - LEADER ELECTION TOOK - 20 MS 2016-09-13 05:09:26,140 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/data/version-2/snapshot.0 2016-09-13 05:09:26,137 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000 2016-09-13 05:09:26,141 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/data/version-2 2016-09-13 05:09:26,141 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 37 MS 2016-09-13 05:09:26,185 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39078:LearnerHandler@382] - Follower sid: 0 : info : 127.0.0.1:13916:13917:participant;127.0.0.1:13915 2016-09-13 05:09:26,191 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39079:LearnerHandler@382] - Follower sid: 1 : info : 127.0.0.1:13919:13920:participant;127.0.0.1:13918 2016-09-13 05:09:26,198 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39079:LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0 2016-09-13 05:09:26,198 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39079:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 1 2016-09-13 05:09:26,198 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39078:LearnerHandler@683] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0 2016-09-13 05:09:26,199 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39078:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 0 2016-09-13 05:09:26,201 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@366] - Getting a diff from the leader 0x0 2016-09-13 05:09:26,206 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@509] - Learner received NEWLEADER message 2016-09-13 05:09:26,209 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/data/version-2/snapshot.0 2016-09-13 05:09:26,210 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Learner@366] - Getting a diff from the leader 0x0 2016-09-13 05:09:26,211 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Learner@509] - Learner received NEWLEADER message 2016-09-13 05:09:26,213 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@1255] - Have quorum of supporters, sids: [ [1, 2],[1, 2] ]; starting up and setting last processed zxid: 0x100000000 2016-09-13 05:09:26,215 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/data/version-2/snapshot.0 2016-09-13 05:09:26,238 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads. 2016-09-13 05:09:26,252 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000 2016-09-13 05:09:26,254 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Learner@493] - Learner received UPTODATE message 2016-09-13 05:09:26,254 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@493] - Learner received UPTODATE message 2016-09-13 05:09:26,266 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads. 2016-09-13 05:09:26,272 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads. 2016-09-13 05:09:26,283 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915 2016-09-13 05:09:26,284 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38385 2016-09-13 05:09:26,285 [myid:0] - INFO [NIOWorkerThread-3:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:38385 2016-09-13 05:09:26,286 [myid:0] - INFO [NIOWorkerThread-3:StatCommand@49] - Stat command output 2016-09-13 05:09:26,287 [myid:0] - INFO [NIOWorkerThread-3:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38385 (no session established for client) 2016-09-13 05:09:26,288 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13918 2016-09-13 05:09:26,288 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13918:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:55309 2016-09-13 05:09:26,291 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:55309 2016-09-13 05:09:26,291 [myid:1] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output 2016-09-13 05:09:26,292 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:55309 (no session established for client) 2016-09-13 05:09:26,292 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13921 2016-09-13 05:09:26,293 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13921:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45002 2016-09-13 05:09:26,297 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:45002 2016-09-13 05:09:26,297 [myid:2] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output 2016-09-13 05:09:26,298 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:45002 (no session established for client) 2016-09-13 05:09:26,304 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.6.0-SNAPSHOT--1, built on 09/13/2016 05:08 GMT 2016-09-13 05:09:26,304 [myid:] - INFO [main:Environment@109] - Client environment:host.name=jenkins-test-5f3 2016-09-13 05:09:26,304 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_101 2016-09-13 05:09:26,305 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation 2016-09-13 05:09:26,305 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre 2016-09-13 05:09:26,305 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/netty-3.10.5.Final. ...[truncated 31909 chars]... 3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 3, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900) 2016-09-13 05:09:26,894 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:26,894 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820) 2016-09-13 05:09:26,894 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 3 2016-09-13 05:09:26,895 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@915] - Connection broken for id 3, my id = 1, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900) 2016-09-13 05:09:26,895 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:26,895 [myid:1] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820) 2016-09-13 05:09:26,896 [myid:1] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 1 2016-09-13 05:09:26,905 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 0 2016-09-13 05:09:26,905 [myid:0] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@915] - Connection broken for id 3, my id = 0, error = java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900) 2016-09-13 05:09:26,906 [myid:0] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:26,905 [myid:3] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 3, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900) 2016-09-13 05:09:26,906 [myid:3] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker 2016-09-13 05:09:26,905 [myid:0] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820) 2016-09-13 05:09:26,913 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Learner@417] - Got zxid 0x100000002 expected 0x1 2016-09-13 05:09:26,917 [myid:3] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820) 2016-09-13 05:09:26,918 [myid:3] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 3 2016-09-13 05:09:26,917 [myid:0] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 0 2016-09-13 05:09:26,917 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13920 2016-09-13 05:09:26,919 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1481] - writeToDisk == true but configFilename == null 2016-09-13 05:09:26,920 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1417] - Restarting Leader Election 2016-09-13 05:09:26,920 [myid:3] - INFO [/127.0.0.1:13926:QuorumCnxManager$Listener@661] - Leaving listener 2016-09-13 05:09:26,922 [myid:0] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13917 2016-09-13 05:09:26,936 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Learner@493] - Learner received UPTODATE message 2016-09-13 05:09:26,936 [myid:3] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13926 2016-09-13 05:09:27,036 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads. 2016-09-13 05:09:27,061 [myid:3] - INFO [SyncThread:3:FileTxnLog@204] - Creating new log file: log.100000002 2016-09-13 05:09:28,635 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13924. Will not attempt to authenticate using SASL (unknown error) 2016-09-13 05:09:28,635 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:50397, server: 127.0.0.1/127.0.0.1:13924 2016-09-13 05:09:28,636 [myid:3] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:13924:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:50397 2016-09-13 05:09:28,651 [myid:3] - INFO [NIOWorkerThread-2:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:50397 2016-09-13 05:09:28,652 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Follower@122] - Got zxid 0x100000003 expected 0x1 2016-09-13 05:09:28,654 [myid:3] - INFO [CommitProcWorkThread-1:ZooKeeperServer@709] - Established session 0x3000048422c0000 with negotiated timeout 30000 for client /127.0.0.1:50397 2016-09-13 05:09:28,661 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:13924, sessionid = 0x3000048422c0000, negotiated timeout = 30000 2016-09-13 05:09:28,690 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED testDuringLeaderSync java.lang.AssertionError: zoo.cfg.dynamic.next is not deleted. at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:155) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) 2016-09-13 05:09:28,692 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled) 2016-09-13 05:09:28,692 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:395) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:60) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:257) at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.tearDown(ReconfigDuringLeaderSyncTest.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) 2016-09-13 05:09:28,693 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=Follower,name3=Connections,name4=127.0.0.1,name5=0x483fd50000] 2016-09-13 05:09:28,694 [myid:] - INFO [main:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38389 which had sessionid 0x483fd50000 2016-09-13 05:09:28,694 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-09-13 05:09:28,694 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down 2016-09-13 05:09:28,694 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-09-13 05:09:28,695 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x483fd50000, likely server has closed socket, closing socket connection and attempting reconnect 2016-09-13 05:09:28,695 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-09-13 05:09:28,695 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-09-13 05:09:28,696 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@299] - CommitProcessor exited loop! 2016-09-13 05:09:28,696 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-09-13 05:09:28,697 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=Follower,name3=InMemoryDataTree] 2016-09-13 05:09:28,697 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-09-13 05:09:28,697 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-09-13 05:09:28,698 [myid:0] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-09-13 05:09:28,698 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-09-13 05:09:28,699 [myid:0] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-09-13 05:09:28,699 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@661] - Leaving listener 2016-09-13 05:09:28,700 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled) 2016-09-13 05:09:28,700 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled) to exit thread 2016-09-13 05:09:29,738 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-09-13 05:09:29,738 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-09-13 05:09:29,776 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-09-13 05:09:29,785 [myid:3] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-09-13 05:09:29,785 [myid:1] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-09-13 05:09:29,789 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-09-13 05:09:29,790 [myid:2] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-09-13 05:09:29,792 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-09-13 05:09:29,922 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-09-13 05:09:29,923 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down 2016-09-13 05:09:30,255 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=Follower] 2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-09-13 05:09:30,256 [myid:0] - WARN [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-09-13 05:09:30,256 [myid:0] - WARN [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0] 2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0] 2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.1] 2016-09-13 05:09:30,257 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.2] 2016-09-13 05:09:30,257 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.3] 2016-09-13 05:09:30,258 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled) 2016-09-13 05:09:30,258 [myid:] - INFO [main:Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:395) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:60) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:257) at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.tearDown(ReconfigDuringLeaderSyncTest.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) 2016-09-13 05:09:30,259 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down 2016-09-13 05:09:30,259 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down 2016-09-13 05:09:30,259 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down 2016-09-13 05:09:30,259 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-09-13 05:09:30,259 [myid:1] - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-09-13 05:09:30,259 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-09-13 05:09:30,259 [myid:1] - INFO [CommitProcessor:1:CommitProcessor@299] - CommitProcessor exited loop! 2016-09-13 05:09:30,260 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower,name3=InMemoryDataTree] 2016-09-13 05:09:30,260 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-09-13 05:09:30,260 [myid:1] - INFO [SyncThread:1:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-09-13 05:09:30,261 [myid:1] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-09-13 05:09:30,261 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13918:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-09-13 05:09:30,261 [myid:1] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-09-13 05:09:30,261 [myid:1] - INFO [/127.0.0.1:13920:QuorumCnxManager$Listener@661] - Leaving listener 2016-09-13 05:09:30,261 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled) 2016-09-13 05:09:30,262 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled) to exit thread 2016-09-13 05:09:30,624 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13915. Will not attempt to authenticate using SASL (unknown error) 2016-09-13 05:09:30,624 [myid:127.0.0.1:13915] - WARN [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1235] - Session 0x483fd50000 for server 127.0.0.1/127.0.0.1:13915, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) 2016-09-13 05:09:31,733 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13915. Will not attempt to authenticate using SASL (unknown error) 2016-09-13 05:09:31,734 [myid:127.0.0.1:13915] - WARN [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1235] - Session 0x483fd50000 for server 127.0.0.1/127.0.0.1:13915, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) 2016-09-13 05:09:32,255 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower] 2016-09-13 05:09:32,256 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-09-13 05:09:32,256 [myid:1] - WARN [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-09-13 05:09:32,256 [myid:1] - WARN [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-09-13 05:09:32,256 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1] 2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1] 2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.0] 2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.2] 2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.3] 2016-09-13 05:09:32,257 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled) 2016-09-13 05:09:32,258 [myid:] - INFO [main:Leader@623] - Shutting down 2016-09-13 05:09:32,258 [myid:] - INFO [main:Leader@629] - Shutdown called java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:629) at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1181) at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:395) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:60) at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:257) at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.tearDown(ReconfigDuringLeaderSyncTest.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) 2016-09-13 05:09:32,259 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down 2016-09-13 05:09:32,259 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down 2016-09-13 05:09:32,259 [myid:] - INFO [main:LeaderRequestProcessor@77] - Shutting down 2016-09-13 05:09:32,259 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down 2016-09-13 05:09:32,259 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down 2016-09-13 05:09:32,259 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop! 2016-09-13 05:09:32,259 [myid:] - INFO [main:CommitProcessor@414] - Shutting down 2016-09-13 05:09:32,260 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@299] - CommitProcessor exited loop! 2016-09-13 05:09:32,260 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@924] - Shutting down 2016-09-13 05:09:32,260 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete 2016-09-13 05:09:32,261 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down 2016-09-13 05:09:32,262 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-09-13 05:09:32,262 [myid:2] - INFO [LearnerCnxAcceptor-/127.0.0.1:13922:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.SocketException: Socket closed 2016-09-13 05:09:32,263 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=InMemoryDataTree] 2016-09-13 05:09:32,264 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39078:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:39078 ******** 2016-09-13 05:09:32,265 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Follower@93] - Exception when following the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:155) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:89) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111) 2016-09-13 05:09:32,265 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39079:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:39079 ******** 2016-09-13 05:09:32,265 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39104:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:39104 ******** 2016-09-13 05:09:32,265 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower] 2016-09-13 05:09:32,265 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39104:LearnerHandler@903] - Ignoring unexpected exception java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622) 2016-09-13 05:09:32,265 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2016-09-13 05:09:32,266 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13921:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-09-13 05:09:32,266 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=Connections,name4=127.0.0.1,name5=0x3000048422c0000] 2016-09-13 05:09:32,266 [myid:2] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-09-13 05:09:32,266 [myid:2] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-09-13 05:09:32,266 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:50397 which had sessionid 0x3000048422c0000 2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):LearnerZooKeeperServer@165] - Shutting down 2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):ZooKeeperServer@529] - shutting down 2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FollowerRequestProcessor@138] - Shutting down 2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):CommitProcessor@414] - Shutting down 2016-09-13 05:09:32,267 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x3000048422c0000, likely server has closed socket, closing socket connection and attempting reconnect 2016-09-13 05:09:32,268 [myid:3] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop! 2016-09-13 05:09:32,268 [myid:3] - INFO [CommitProcessor:3:CommitProcessor@299] - CommitProcessor exited loop! 2016-09-13 05:09:32,268 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FinalRequestProcessor@479] - shutdown of request processor complete 2016-09-13 05:09:32,269 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=InMemoryDataTree] 2016-09-13 05:09:32,269 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):SyncRequestProcessor@191] - Shutting down 2016-09-13 05:09:32,269 [myid:3] - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited! 2016-09-13 05:09:32,269 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-09-13 05:09:32,269 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1033] - LOOKING 2016-09-13 05:09:32,270 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1465497725988500626.junit.dir/data/version-2/snapshot.100000001 2016-09-13 05:09:32,271 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader] 2016-09-13 05:09:32,272 [myid:2] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1127] - Unexpected exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:561) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1124) 2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@623] - Shutting down 2016-09-13 05:09:32,272 [myid:2] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING 2016-09-13 05:09:32,272 [myid:2] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2] 2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2] 2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.0] 2016-09-13 05:09:32,273 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.1] 2016-09-13 05:09:32,273 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.3] 2016-09-13 05:09:32,274 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@661] - Leaving listener 2016-09-13 05:09:32,274 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled) 2016-09-13 05:09:32,274 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled) to exit thread 2016-09-13 05:09:32,274 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled) 2016-09-13 05:09:32,275 [myid:3] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted 2016-09-13 05:09:32,275 [myid:3] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:13924:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method 2016-09-13 05:09:32,276 [myid:3] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method 2016-09-13 05:09:32,276 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x100000004 2016-09-13 05:09:32,277 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 0 at election address /127.0.0.1:13917 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:32,277 [myid:3] - INFO [/127.0.0.1:13926:QuorumCnxManager$Listener@661] - Leaving listener 2016-09-13 05:09:32,277 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 0 at election address /127.0.0.1:13917 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:489) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:32,278 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 1 at election address /127.0.0.1:13920 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:32,278 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled) 2016-09-13 05:09:32,279 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled) to exit thread 2016-09-13 05:09:32,279 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 1 at election address /127.0.0.1:13920 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:489) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465) at java.lang.Thread.run(Thread.java:745) 2016-09-13 05:09:32,280 [myid:3] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down 2016-09-13 05:09:32,280 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=LeaderElection] 2016-09-13 05:09:32,280 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited 2016-09-13 05:09:32,280 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3] 2016-09-13 05:09:32,280 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3] 2016-09-13 05:09:32,281 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.0] 2016-09-13 05:09:32,281 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.1] 2016-09-13 05:09:32,281 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.2] 2016-09-13 05:09:32,283 [myid:] - INFO [main:ZKTestCase$1@70] - FAILED testDuringLeaderSync java.lang.AssertionError: zoo.cfg.dynamic.next is not deleted. at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:155) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) 2016-09-13 05:09:32,284 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testDuringLeaderSync {noformat}

    Apache's JIRA Issue Tracker | 3 months ago | Michael Han
    java.lang.Exception: shutdown Follower
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    exception causing shutdownthere are 5 members in the quorum. one follower can not sync with leader after restart. it seems leader has close the data connection with this follower because of read timeout. here is the key log in follower: {noformat} 2011-06-30 22:14:45,069 - WARN [Thread-17:QuorumCnxManager$RecvWorker@658] - Connection broken: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:113) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:156) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) 2011-06-30 22:14:45,069 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@689] - Notification: 3, 17198470148, 3, 3, LOOKING, LOOKING, 3 2011-06-30 22:14:45,070 - ERROR [Thread-16:QuorumCnxManager$SendWorker@559] - Failed to send last message. Shutting down thread. java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:557) 2011-06-30 22:14:45,082 - INFO [QuorumPeer:/0.0.0.0:2181:Learner@282] - Getting a diff from the leader 0x4011bd462 2011-06-30 22:14:45,083 - WARN [Thread-18:QuorumCnxManager$SendWorker@589] - Send worker leaving thread 2011-06-30 22:14:45,085 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@116] - Got zxid 0x4011bd405 expected 0x1 2011-06-30 22:14:45,090 - INFO [QuorumPeer:/0.0.0.0:2181:FileTxnSnapLog@208] - Snapshotting: 4011bd462 2011-06-30 22:14:53,397 - WARN [SyncThread:3:SendAckRequestProcessor@63] - Closing connection to leader, exception during packet send java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:126) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:61) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:164) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98) 2011-06-30 22:14:53,398 - WARN [QuorumPeer:/0.0.0.0:2181:Follower@82] - Exception when following the leader java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:126) at org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:358) at org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:108) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:634) 2011-06-30 22:14:53,398 - WARN [SyncThread:3:SendAckRequestProcessor@63] - Closing connection to leader, exception during packet send java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:126) at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:61) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:164) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98) 2011-06-30 22:14:53,399 - INFO [QuorumPeer:/0.0.0.0:2181:Follower@166] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:638) and these are the leader's: 2011-06-30 22:14:35,943 - ERROR [LearnerHandler-/10.23.247.163:14975:LearnerHandler@444] - Unexpected exception causing shutdown while sock still open java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:358) 2011-06-30 22:14:35,943 - WARN [LearnerHandler-/10.23.247.163:14975:LearnerHandler@457] - ******* GOODBYE /10.23.247.163:14975 ******** 2011-06-30 22:14:48,943 - ERROR [CommitProcessor:4:NIOServerCnxn@422] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:395) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1360) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:367) at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:535) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) 2011-06-30 22:14:49,084 - ERROR [LearnerHandler-/10.23.247.163:14998:LearnerHandler@444] - Unexpected exception causing shutdown while sock still open java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:358) 2011-06-30 22:14:49,084 - WARN [LearnerHandler-/10.23.247.163:14998:LearnerHandler@457] - ******* GOODBYE /10.23.247.163:14998 ******** {noformat}

    Apache's JIRA Issue Tracker | 5 years ago | helei
    java.lang.Exception: shutdown Follower
  6. 0

    I'd like to setup 3-nodes zookeeper cluster with version 3.5.1. Because the network model of kubernetes is that pod and service has different ip address. In order to deploy it into kubernetes, I have to make zookeeper pods itself to be 0.0.0.0, so it can be started correctly. the configuration as below: zk1: zoo.cfg standaloneEnabled=false dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic zoo.cfg.dynamic server.1=0.0.0.0:2888:3888:participant;2181 server.2=10.62.56.192:2888:3888:participant;2181 server.3=10.62.56.193:2888:3888:participant;2181 zk2: zoo.cfg standaloneEnabled=false dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic zoo.cfg.dynamic server.1=10.62.56.191:2888:3888:participant;2181 server.2=0.0.0.0:2888:3888:participant;2181 server.3=10.62.56.193:2888:3888:participant;2181 zk3: zoo.cfg standaloneEnabled=false dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic zoo.cfg.dynamic server.1=10.62.56.191:2888:3888:participant;2181 server.2=10.62.56.192:2888:3888:participant;2181 server.3=0.0.0.0:2888:3888:participant;218 The result is that: 1. Looks like the election is successful. a new dynamic file is generated in every node:/opt/zookeeper/conf/zoo.cfg.dynamic.100000000 like below server.1=10.62.56.191:2888:3888:participant;0.0.0.0:2181 server.2=0.0.0.0:2888:3888:participant;0.0.0.0:2181 server.3=10.62.56.193:2888:3888:participant;0.0.0.0:2181 2. But the cluster doesn't really work, I saw the errors: 0:0:2181)(secure=disabled):Learner@273] - Unexpected exception, tries=3, remaining init limit=16997, connecting to /0.0.0.0:2888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227) at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:74) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064) 2015-12-15 04:35:00,403 [myid:1] - INFO 2015-12-15 04:35:00,585 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)

    Apache's JIRA Issue Tracker | 12 months ago | cheyang
    java.lang.Exception: shutdown Follower

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.Exception

      shutdown Follower

      at org.apache.zookeeper.server.quorum.Follower.shutdown()
    2. Zookeeper
      QuorumPeer.shutdown
      1. org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
      2. org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
      2 frames
    3. org.apache.zookeeper
      QuorumZxidSyncTest.tearDown
      1. org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
      2. org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:300)
      3. org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
      4. org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
      4 frames
    4. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:606)
      4 frames
    5. JUnit
      JUnit4TestAdapter.run
      1. org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
      2. org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      3. org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
      4. org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
      5. org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
      6. org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
      7. org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      8. org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
      9. org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
      10. org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
      11. org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
      12. org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
      13. org.junit.runners.ParentRunner.run(ParentRunner.java:236)
      14. junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
      14 frames
    6. Apache Ant + JUnit
      JUnitTestRunner.main
      1. org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
      2. org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
      3. org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
      3 frames