java.io.IOException: No space left on device

Apache's JIRA Issue Tracker | suja s | 4 years ago
  1. 0

    Once zookeeper data dir disk becomes full, the process gets shut down. {noformat} 2012-12-14 13:22:26,959 [myid:2] - ERROR [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@276] - Severe unrecoverable error, exiting java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:56) at java.io.DataOutputStream.write(DataOutputStream.java:90) at java.io.FilterOutputStream.write(FilterOutputStream.java:80) at org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119) at org.apache.zookeeper.server.DataNode.serialize(DataNode.java:168) at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1115) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1130) at org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1130) at org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1179) at org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:138) at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:213) at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:230) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:242) at org.apache.zookeeper.server.ZooKeeperServer.takeSnapshot(ZooKeeperServer.java:274) at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:407) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:82) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:759) {noformat} Later disk space is cleared and zk started again. Startup of zk fails as it is not able to read snapshot properly. (Since load from disk failed it is not able to join peers in the quorum and get a snapshot diff) {noformat} 2012-12-14 16:20:31,489 [myid:2] - INFO [main:FileSnap@83] - Reading snapshot ../dataDir/version-2/snapshot.1000000042 2012-12-14 16:20:31,564 [myid:2] - ERROR [main:QuorumPeer@472] - Unable to load database on disk java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:529) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:504) at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:341) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:436) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:428) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:152) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) 2012-12-14 16:20:31,566 [myid:2] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:473) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:428) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:152) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:529) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:504) at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:341) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132) {noformat}

    Apache's JIRA Issue Tracker | 4 years ago | suja s
    java.io.IOException: No space left on device
  2. 0

    Unable to recover for 1 failing node in 3 nodes ensemble

    zookeeper-user | 2 years ago | Benjamin Jaton
    java.io.IOException: No space left on device
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Unable to recover for 1 failing node in 3 nodes ensemble

    zookeeper-user | 2 years ago | Benjamin Jaton
    java.io.IOException: No space left on device
  5. 0

    GZIP doesnt work when rest controller return a callable

    GitHub | 2 years ago | tofmonaute
    java.io.IOException: write beyond end of stream

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      No space left on device

      at java.io.FileOutputStream.writeBytes()
    2. Java RT
      FilterOutputStream.write
      1. java.io.FileOutputStream.writeBytes(Native Method)
      2. java.io.FileOutputStream.write(FileOutputStream.java:282)
      3. java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
      4. java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
      5. java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:56)
      6. java.io.DataOutputStream.write(DataOutputStream.java:90)
      7. java.io.FilterOutputStream.write(FilterOutputStream.java:80)
      7 frames
    3. Zookeeper
      QuorumPeer.run
      1. org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119)
      2. org.apache.zookeeper.server.DataNode.serialize(DataNode.java:168)
      3. org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
      4. org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1115)
      5. org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1130)
      6. org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1130)
      7. org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1179)
      8. org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:138)
      9. org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:213)
      10. org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:230)
      11. org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:242)
      12. org.apache.zookeeper.server.ZooKeeperServer.takeSnapshot(ZooKeeperServer.java:274)
      13. org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:407)
      14. org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:82)
      15. org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:759)
      15 frames