java.io.IOException: read past EOF

JIRA | Nico Krijnen | 6 years ago
  1. 0

    We have a system running lucene and zoie. We use lucene as a content store for a CMS/DAM system. We use the hot-backup feature of zoie to make scheduled backups of the index. This works fine for small indexes and when there are not a lot of changes to the index when the backup is made. On large indexes (about 5 GB to 19 GB), when a backup is made while the index is being changed a lot (lots of document additions and/or deletions), we almost always get a 'read past EOF' at some point, followed by lots of 'Lock obtain timed out'. At that point we get lots of 0 kb files in the index, data gets lots, and the index is unusable. When we stop our server, remove the 0kb files and restart our server, the index is operational again, but data has been lost. I'm not sure if this is a zoie or a lucene issue, so i'm posting it to both. Hopefully someone has some ideas where to look to fix this. Some more details... Stack trace of the read past EOF and following Lock obtain timed out: {code} 78307 [proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader@31ca5085] ERROR proj.zoie.impl.indexing.internal.BaseSearchIndex - read past EOF java.io.IOException: read past EOF at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:154) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39) at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:37) at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:69) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:245) at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:166) at org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:725) at org.apache.lucene.index.IndexReader.commit(IndexReader.java:987) at org.apache.lucene.index.IndexReader.commit(IndexReader.java:973) at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:162) at org.apache.lucene.index.IndexReader.close(IndexReader.java:1003) at proj.zoie.impl.indexing.internal.BaseSearchIndex.deleteDocs(BaseSearchIndex.java:203) at proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:223) at proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153) at proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134) at proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171) at proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373) 579336 [proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader@31ca5085] ERROR proj.zoie.impl.indexing.internal.LuceneIndexDataLoader - Problem copying segments: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@5ad0b895: write.lock org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@5ad0b895: write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1060) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:957) at proj.zoie.impl.indexing.internal.DiskSearchIndex.openIndexWriter(DiskSearchIndex.java:176) at proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:228) at proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153) at proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134) at proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171) at proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373) {code} We get exactly the same behavour on both OS X and on Windows. On both zoie is using a SimpleFSDirectory. We also use a SingleInstanceLockFactory (since our process is the only one working with the index), but we get the same behaviour with a NativeFSLock. The snapshot backup is being made by calling: *proj.zoie.impl.indexing.ZoieSystem.exportSnapshot(WritableByteChannel)* Same issue in lucene JIRA: https://issues.apache.org/jira/browse/LUCENE-2729

    JIRA | 6 years ago | Nico Krijnen
    java.io.IOException: read past EOF
  2. 0

    We have a system running lucene and zoie. We use lucene as a content store for a CMS/DAM system. We use the hot-backup feature of zoie to make scheduled backups of the index. This works fine for small indexes and when there are not a lot of changes to the index when the backup is made. On large indexes (about 5 GB to 19 GB), when a backup is made while the index is being changed a lot (lots of document additions and/or deletions), we almost always get a 'read past EOF' at some point, followed by lots of 'Lock obtain timed out'. At that point we get lots of 0 kb files in the index, data gets lots, and the index is unusable. When we stop our server, remove the 0kb files and restart our server, the index is operational again, but data has been lost. I'm not sure if this is a zoie or a lucene issue, so i'm posting it to both. Hopefully someone has some ideas where to look to fix this. Some more details... Stack trace of the read past EOF and following Lock obtain timed out: {code} 78307 [proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader@31ca5085] ERROR proj.zoie.impl.indexing.internal.BaseSearchIndex - read past EOF java.io.IOException: read past EOF at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:154) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39) at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:37) at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:69) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:245) at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:166) at org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:725) at org.apache.lucene.index.IndexReader.commit(IndexReader.java:987) at org.apache.lucene.index.IndexReader.commit(IndexReader.java:973) at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:162) at org.apache.lucene.index.IndexReader.close(IndexReader.java:1003) at proj.zoie.impl.indexing.internal.BaseSearchIndex.deleteDocs(BaseSearchIndex.java:203) at proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:223) at proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153) at proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134) at proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171) at proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373) 579336 [proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader@31ca5085] ERROR proj.zoie.impl.indexing.internal.LuceneIndexDataLoader - Problem copying segments: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@5ad0b895: write.lock org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@5ad0b895: write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1060) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:957) at proj.zoie.impl.indexing.internal.DiskSearchIndex.openIndexWriter(DiskSearchIndex.java:176) at proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:228) at proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153) at proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134) at proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171) at proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373) {code} We get exactly the same behavour on both OS X and on Windows. On both zoie is using a SimpleFSDirectory. We also use a SingleInstanceLockFactory (since our process is the only one working with the index), but we get the same behaviour with a NativeFSLock. The snapshot backup is being made by calling: *proj.zoie.impl.indexing.ZoieSystem.exportSnapshot(WritableByteChannel)* Same issue in lucene JIRA: https://issues.apache.org/jira/browse/LUCENE-2729

    JIRA | 6 years ago | Nico Krijnen
    java.io.IOException: read past EOF
  3. 0

    Solr indexing - Master/Slave replication, how to handle huge index and high traffic?

    Stack Overflow | 6 years ago | Fanny H.
    org.apache.solr.common.SolrException: Index fetch failed :
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    java.io.IOException: read past EOF Lucene in Java with Eclipse

    Stack Overflow | 1 year ago | hugh
    java.io.IOException: read past EOF
  6. 0

    We had a case where Nexus was stopped (not cleanly shutdown) due to a physical machine reboot. When trying to bring the service back up the exception below was thrown in the logs. The resolution was to delete the Lucene index at /sonatype-work/nexus/timeline and then the server started correctly. 2009-03-25 11:05:53.743 ERROR [WrapperListener_start_runner:] - org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@ee260b{/repo,../../../runtime/apps/nexus/webapp} java.lang.IllegalStateException: Could not initialize Nexus. at org.sonatype.nexus.web.NexusBooterListener.contextInitialized(NexusBooterListener.java:45) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1234) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:460) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:222) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.sonatype.plexus.jetty.DefaultServletContainer.start(DefaultServletContainer.java:312) at org.codehaus.plexus.personality.plexus.lifecycle.phase.StartPhase.execute(StartPhase.java:33) at org.codehaus.plexus.lifecycle.AbstractLifecycleHandler.start(AbstractLifecycleHandler.java:97) at org.codehaus.plexus.component.builder.XBeanComponentBuilder.startComponentLifecycle(XBeanComponentBuilder.java:220) at org.codehaus.plexus.component.builder.XBeanComponentBuilder.build(XBeanComponentBuilder.java:98) at org.codehaus.plexus.component.manager.AbstractComponentManager.createComponentInstance(AbstractComponentManager.java:135) at org.codehaus.plexus.component.manager.ClassicSingletonComponentManager.getComponent(ClassicSingletonComponentManager.java:96) at org.codehaus.plexus.DefaultComponentLookupManager.lookup(DefaultComponentLookupManager.java:147) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:484) at org.codehaus.plexus.container.initialization.StartLoadOnStartComponentsPhase.execute(StartLoadOnStartComponentsPhase.java:69) at org.codehaus.plexus.DefaultPlexusContainer.initializePhases(DefaultPlexusContainer.java:849) at org.codehaus.plexus.DefaultPlexusContainer.initialize(DefaultPlexusContainer.java:813) at org.codehaus.plexus.DefaultPlexusContainer.construct(DefaultPlexusContainer.java:405) at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:196) at org.sonatype.appbooter.PlexusContainerHost.startContainer(PlexusContainerHost.java:280) at org.sonatype.appbooter.jsw.PlexusJSWListener.start(PlexusJSWListener.java:99) at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788) Caused by: org.codehaus.plexus.component.repository.exception.ComponentLookupException: Unable to lookup component 'org.sonatype.nexus.Nexus', it could not be started. role: org.sonatype.nexus.Nexus roleHint: default classRealm: plexus at org.codehaus.plexus.DefaultComponentLookupManager.lookup(DefaultComponentLookupManager.java:155) at org.codehaus.plexus.DefaultComponentLookupManager.lookup(DefaultComponentLookupManager.java:69) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:430) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:424) at org.sonatype.nexus.web.NexusBooterListener.contextInitialized(NexusBooterListener.java:39) ... 29 more Caused by: org.codehaus.plexus.component.repository.exception.ComponentLifecycleException: Error starting component at org.codehaus.plexus.component.builder.XBeanComponentBuilder.startComponentLifecycle(XBeanComponentBuilder.java:222) at org.codehaus.plexus.component.builder.XBeanComponentBuilder.build(XBeanComponentBuilder.java:98) at org.codehaus.plexus.component.manager.AbstractComponentManager.createComponentInstance(AbstractComponentManager.java:135) at org.codehaus.plexus.component.manager.ClassicSingletonComponentManager.getComponent(ClassicSingletonComponentManager.java:96) at org.codehaus.plexus.DefaultComponentLookupManager.lookup(DefaultComponentLookupManager.java:147) ... 33 more Caused by: org.codehaus.plexus.personality.plexus.lifecycle.phase.PhaseExecutionException: Error starting component at org.codehaus.plexus.personality.plexus.lifecycle.phase.StartPhase.execute(StartPhase.java:37) at org.codehaus.plexus.lifecycle.AbstractLifecycleHandler.start(AbstractLifecycleHandler.java:97) at org.codehaus.plexus.component.builder.XBeanComponentBuilder.startComponentLifecycle(XBeanComponentBuilder.java:220) ... 37 more Caused by: org.codehaus.plexus.personality.plexus.lifecycle.phase.StartingException: Could not start Nexus! at org.sonatype.nexus.DefaultNexus.start(DefaultNexus.java:1564) at org.codehaus.plexus.personality.plexus.lifecycle.phase.StartPhase.execute(StartPhase.java:33) ... 39 more Caused by: org.codehaus.plexus.personality.plexus.lifecycle.phase.StartingException: Cannot start Timeline! at org.sonatype.nexus.timeline.DefaultTimeline.startService(DefaultTimeline.java:173) at org.sonatype.nexus.feeds.DefaultFeedRecorder.startService(DefaultFeedRecorder.java:101) at org.sonatype.nexus.DefaultNexus.startService(DefaultNexus.java:1600) at org.sonatype.nexus.DefaultNexus.start(DefaultNexus.java:1560) ... 40 more Caused by: java.io.IOException: read past EOF at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:146) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:66) at org.apache.lucene.store.IndexInput.readLong(IndexInput.java:89) at org.apache.lucene.index.SegmentInfo.<init>(SegmentInfo.java:140) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:220) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:254) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:636) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:251) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:710) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:614) at org.sonatype.nexus.index.context.NexusIndexWriter.<init>(NexusIndexWriter.java:35) at org.sonatype.nexus.timeline.DefaultTimeline.startService(DefaultTimeline.java:155) ... 43 more

    Sonatype JIRA | 8 years ago | Eric Dalquist
    java.lang.IllegalStateException: Could not initialize Nexus.

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      read past EOF

      at org.apache.lucene.store.BufferedIndexInput.refill()
    2. Lucene
      IndexReader.close
      1. org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:154)
      2. org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
      3. org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:37)
      4. org.apache.lucene.store.IndexInput.readInt(IndexInput.java:69)
      5. org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:245)
      6. org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:166)
      7. org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:725)
      8. org.apache.lucene.index.IndexReader.commit(IndexReader.java:987)
      9. org.apache.lucene.index.IndexReader.commit(IndexReader.java:973)
      10. org.apache.lucene.index.IndexReader.decRef(IndexReader.java:162)
      11. org.apache.lucene.index.IndexReader.close(IndexReader.java:1003)
      11 frames
    3. proj.zoie.impl
      BatchedIndexDataLoader$LoaderThread.run
      1. proj.zoie.impl.indexing.internal.BaseSearchIndex.deleteDocs(BaseSearchIndex.java:203)
      2. proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:223)
      3. proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153)
      4. proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134)
      5. proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171)
      6. proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373)
      6 frames