java.io.IOException: The accepted epoch, 1 is less than the current epoch, 2

Apache's JIRA Issue Tracker | Ramnatthan Alagappan | 3 months ago
  1. 0

    I am running a three node ZooKeeper cluster. Renames of acceptedEpoch.tmp to acceptedEpoch and currentEpoch.tmp to currentEpoch have to persisted to disk by explicitly issuing fsync on the parent directory. If not, the rename might not hit the disk immediately and if a crash occurs at this point, then the server would fail to start with the following error in the log. If this happens on two more or nodes, then the cluster can become unavailable. [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /tmp/zoo2.cfg [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.2 to address: /127.0.0.2 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.4 to address: /127.0.0.4 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.3 to address: /127.0.0.3 [myid:] - INFO [main:QuorumPeerConfig@331] - Defaulting to majority quorums [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. [myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer [myid:1] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2182 [myid:1] - INFO [main:QuorumPeer@1019] - tickTime set to 2000 [myid:1] - INFO [main:QuorumPeer@1039] - minSessionTimeout set to -1 [myid:1] - INFO [main:QuorumPeer@1050] - maxSessionTimeout set to -1 [myid:1] - INFO [main:QuorumPeer@1065] - initLimit set to 5 [myid:1] - INFO [main:FileSnap@83] - Reading snapshot /run/shm/dice-4636/113-98-129-z_majority_RO_OM_0=60_1=55/rdir-0/version-2/snapshot.100000002 [myid:1] - ERROR [main:QuorumPeer@557] - Unable to load database on disk java.io.IOException: The accepted epoch, 1 is less than the current epoch, 2 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:554) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) 2016-04-15 03:24:57,144 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Caused by: java.io.IOException: The accepted epoch, 1 is less than the current epoch, 2 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:554) ... 4 more Similarly, when new log file is created, the parent directory needs be explicitly fsynced to persist the log file. Otherwise a data loss might be possible (We have reproduced the above issues). Please see this: https://www.quora.com/Linux/When-should-you-fsync-the-containing-directory-in-addition-to-the-file-itself and http://research.cs.wisc.edu/wind/Publications/alice-osdi14.pdf.

    Apache's JIRA Issue Tracker | 3 months ago | Ramnatthan Alagappan
    java.io.IOException: The accepted epoch, 1 is less than the current epoch, 2
  2. 0

    I have zookeeper running normally just fine in a 3-server cluster. Then I try to configure zookeeper to use Kerberos, following docs in the Solr wiki here: https://cwiki.apache.org/confluence/display/solr/Kerberos+Authentication+Plugin I can't even get to the fun Kerberos errors. When I start with {{JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/jaas-server.conf"}} and this jaas-server.conf: {code} Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab=/keytabs/vdev-solr-01.keytab storeKey=true doNotPrompt=true useTicketCache=false debug=true principal="HTTP/<snip>"; } {code} I get this in the log: {code} 2016-02-10 16:16:51,327 [myid:1] - ERROR [main:ServerCnxnFactory@195] - No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'. 2016-02-10 16:16:51,328 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.io.IOException: No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'. at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:196) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) {code} (Note the "foundin" typo.) I get the exact same error if the jaas-server.conf file exists, or does not. So later I found that the Solr wiki was wrong and lost the double quotes around the keytab value. It would be nice if Zookeeper spewed a more useful message when it can't parse the configuration.

    Apache's JIRA Issue Tracker | 10 months ago | Dan Fitch
    java.io.IOException: No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'.
  3. 0

    zookeeper + Kafka - Unable to create data directory

    Stack Overflow | 9 months ago | RookieSpark
    java.io.IOException: Unable to create data directory /var/zookeeper/version-2
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Error Messages to Fear | Hadoop and Kerberos: The Madness Beyond the Gate

    gitbooks.io | 3 months ago
    java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided
  6. 0

    Unable to start zookeeper

    Stack Overflow | 6 months ago | Raja
    java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /search/cluster1/overseer/queue

    4 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      The accepted epoch, 1 is less than the current epoch, 2

      at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase()
    2. Zookeeper
      QuorumPeerMain.main
      1. org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:554)
      2. org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
      3. org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
      4. org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
      5. org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
      5 frames