java.io.IOException: Unreasonable length = -295704495

Apache's JIRA Issue Tracker | Ramnatthan Alagappan | 3 months ago
  1. 0

    I am running a three node ZooKeeper cluster. When a new log file is created by ZooKeeper, I see the following sequence of system calls: 1. creat(new_log) 2. write(new_log, count=16) // This is a log header I believe/ 3. truncate(new_log, from 16 bytes to 16 KBytes) // I have configured the log size to be 16K. When the above sequence of operations complete, it is reasonable to expect the newly created log file to contain the header(16 bytes) and then filled with zeros till the end of the log. But when a crash occurs (due to a power failure), while the truncate system call is in progress, it is possible for the log to contain garbage data when the system restarts from the crash. Note that if the crash occurs just after the truncate system call completes, then there is no problem. Basically, the truncate needs to be atomically persisted for ZooKeeper to recover from crashes correctly or (more realistically) the recovery code needs to deal with the case of expecting garbage in a newly created log. As mentioned, if a crash occurs during the truncate system call, then ZooKeeper will fail to start with the following exception. Here is the stack trace: java.io.IOException: Unreasonable length = -295704495 at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127) at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92) at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:552) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:527) at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:354) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Caused by: java.io.IOException: Unreasonable length = -295704495 at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127) at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92) at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:552) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:527) at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:354) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510) ... 4 more Next, it is possible for two nodes of a 3-node ZooKeeper cluster to reach the same state. In that case, they both will fail to startup, rendering the entire cluster unavailable.

    Apache's JIRA Issue Tracker | 3 months ago | Ramnatthan Alagappan
    java.io.IOException: Unreasonable length = -295704495
  2. 0

    I have zookeeper running normally just fine in a 3-server cluster. Then I try to configure zookeeper to use Kerberos, following docs in the Solr wiki here: https://cwiki.apache.org/confluence/display/solr/Kerberos+Authentication+Plugin I can't even get to the fun Kerberos errors. When I start with {{JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/jaas-server.conf"}} and this jaas-server.conf: {code} Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab=/keytabs/vdev-solr-01.keytab storeKey=true doNotPrompt=true useTicketCache=false debug=true principal="HTTP/<snip>"; } {code} I get this in the log: {code} 2016-02-10 16:16:51,327 [myid:1] - ERROR [main:ServerCnxnFactory@195] - No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'. 2016-02-10 16:16:51,328 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.io.IOException: No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'. at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:196) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) {code} (Note the "foundin" typo.) I get the exact same error if the jaas-server.conf file exists, or does not. So later I found that the Solr wiki was wrong and lost the double quotes around the keytab value. It would be nice if Zookeeper spewed a more useful message when it can't parse the configuration.

    Apache's JIRA Issue Tracker | 10 months ago | Dan Fitch
    java.io.IOException: No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'.
  3. 0

    zookeeper + Kafka - Unable to create data directory

    Stack Overflow | 9 months ago | RookieSpark
    java.io.IOException: Unable to create data directory /var/zookeeper/version-2
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Error Messages to Fear | Hadoop and Kerberos: The Madness Beyond the Gate

    gitbooks.io | 3 months ago
    java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided
  6. 0

    Unable to start zookeeper

    Stack Overflow | 6 months ago | Raja
    java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /search/cluster1/overseer/queue

    4 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Unreasonable length = -295704495

      at org.apache.jute.BinaryInputArchive.checkLength()
    2. Zookeeper
      QuorumPeerMain.main
      1. org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
      2. org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
      3. org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
      4. org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
      5. org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
      6. org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:552)
      7. org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:527)
      8. org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:354)
      9. org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132)
      10. org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
      11. org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
      12. org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
      13. org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
      14. org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
      15. org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
      15 frames