com.atlassian.jira.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException

Atlassian JIRA | Chris Fuller [On Leave until 2016/09/14] | 2 years ago
  1. 0

    [JRA-41409] Lucene index can be corrupted by a graceful shutdown - Atlassian JIRA

    atlassian.com | 1 year ago
    com.atlassian.jira.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
  2. 0

    {{QueuingIndex}} uses thread interruption to implement cancellation. The problem is that there is a race condition if the indexer is closed while an index write operation is in progress, as the interrupt causes the underlying index file channels to abort their write operations, leading to corrupt indexes. See chapter 7 of _Java Concurrency in Practice_ for a detailed description of task cancellation strategies and more information about why using {{interrupt()}} when running other people's code is not a good idea. This is the underlying culprit in JRA-38544. As near as I can tell, it has existed since the indexing code was rewritten in JIRA 4.0. It is normally a very rarely triggered race condition, but especially busy large systems are more likely to encounter it. Example stack trace: {noformat} 2014-12-19 18:07:45,525 some-thread-name WARN some-user [atlassian.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '720151' 2014-12-19 18:07:45,530 some-thread-name WARN some-user [atlassian.jira.index.AccumulatingResultBuilder] java.nio.channels.ClosedByInterruptException com.atlassian.jira.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:137) at com.atlassian.jira.index.DefaultIndexEngine$WriterReference.commit(DefaultIndexEngine.java:230) at com.atlassian.jira.index.DefaultIndexEngine$FlushPolicy$2.commit(DefaultIndexEngine.java:63) at com.atlassian.jira.index.DefaultIndexEngine$FlushPolicy.perform(DefaultIndexEngine.java:88) at com.atlassian.jira.index.DefaultIndexEngine.write(DefaultIndexEngine.java:158) at com.atlassian.jira.index.DefaultIndex.perform(DefaultIndex.java:36) at com.atlassian.jira.index.QueueingIndex$Task.index(QueueingIndex.java:145) at com.atlassian.jira.index.QueueingIndex$Task.run(QueueingIndex.java:130) at java.lang.Thread.run(Thread.java:724) Caused by: java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:880) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:85) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:114) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677) at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249) at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571) at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508) at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410) at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:133) ... 8 more {noformat}

    Atlassian JIRA | 2 years ago | Chris Fuller
    com.atlassian.jira.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
  3. 0

    {{QueuingIndex}} uses thread interruption to implement cancellation. The problem is that there is a race condition if the indexer is closed while an index write operation is in progress, as the interrupt causes the underlying index file channels to abort their write operations, leading to corrupt indexes. See chapter 7 of _Java Concurrency in Practice_ for a detailed description of task cancellation strategies and more information about why using {{interrupt()}} when running other people's code is not a good idea. This is the underlying culprit in JRA-38544. As near as I can tell, it has existed since the indexing code was rewritten in JIRA 4.0. It is normally a very rarely triggered race condition, but especially busy large systems are more likely to encounter it. Example stack trace: {noformat} 2014-12-19 18:07:45,525 some-thread-name WARN some-user [atlassian.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '720151' 2014-12-19 18:07:45,530 some-thread-name WARN some-user [atlassian.jira.index.AccumulatingResultBuilder] java.nio.channels.ClosedByInterruptException com.atlassian.jira.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:137) at com.atlassian.jira.index.DefaultIndexEngine$WriterReference.commit(DefaultIndexEngine.java:230) at com.atlassian.jira.index.DefaultIndexEngine$FlushPolicy$2.commit(DefaultIndexEngine.java:63) at com.atlassian.jira.index.DefaultIndexEngine$FlushPolicy.perform(DefaultIndexEngine.java:88) at com.atlassian.jira.index.DefaultIndexEngine.write(DefaultIndexEngine.java:158) at com.atlassian.jira.index.DefaultIndex.perform(DefaultIndex.java:36) at com.atlassian.jira.index.QueueingIndex$Task.index(QueueingIndex.java:145) at com.atlassian.jira.index.QueueingIndex$Task.run(QueueingIndex.java:130) at java.lang.Thread.run(Thread.java:724) Caused by: java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:880) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:85) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:114) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677) at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249) at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571) at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508) at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410) at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:133) ... 8 more {noformat}

    Atlassian JIRA | 2 years ago | Chris Fuller [On Leave until 2016/09/14]
    com.atlassian.jira.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Recommended way to handle ClosedByInterruptException?

    Google Groups | 4 years ago | rap...@gmail.com
    java.nio.channels.ClosedByInterruptException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.nio.channels.ClosedByInterruptException

      No message provided

      at java.nio.channels.spi.AbstractInterruptibleChannel.end()
    2. Java RT
      FileChannelImpl.map
      1. java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
      2. sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:880)
      2 frames
    3. Lucene
      IndexWriter.commit
      1. org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
      2. org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
      3. org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:85)
      4. org.apache.lucene.index.SegmentReader.get(SegmentReader.java:114)
      5. org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702)
      6. org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677)
      7. org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249)
      8. org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571)
      9. org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508)
      10. org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371)
      11. org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444)
      12. org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426)
      13. org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410)
      13 frames
    4. com.atlassian.jira
      QueueingIndex$Task.run
      1. com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:133)
      2. com.atlassian.jira.index.DefaultIndexEngine$WriterReference.commit(DefaultIndexEngine.java:230)
      3. com.atlassian.jira.index.DefaultIndexEngine$FlushPolicy$2.commit(DefaultIndexEngine.java:63)
      4. com.atlassian.jira.index.DefaultIndexEngine$FlushPolicy.perform(DefaultIndexEngine.java:88)
      5. com.atlassian.jira.index.DefaultIndexEngine.write(DefaultIndexEngine.java:158)
      6. com.atlassian.jira.index.DefaultIndex.perform(DefaultIndex.java:36)
      7. com.atlassian.jira.index.QueueingIndex$Task.index(QueueingIndex.java:145)
      8. com.atlassian.jira.index.QueueingIndex$Task.run(QueueingIndex.java:130)
      8 frames
    5. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:724)
      1 frame