org.apache.cassandra.io.sstable.CorruptSSTableException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Recently we encountered the following situation in production. Client tool performing reading data from some column family failed with System.TimeoutException at the time when there was no any significant load on the cluster. We were able to reproduce the error several times running this same tool on different machines at different times. Client error was: {code} INFO 2014-11-12 17:26:50,217 System.TimeoutException: The task didn't complete before timeout. at Cassandra.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout) in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\TaskHelper.cs:line 137 at Cassandra.RequestHandler`1.<FillRowSet>b__0(Byte[] pagingState) in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RequestHandler.cs:line 110 at Cassandra.RowSet.PageNext() in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RowPopulators\RowSet.cs:line 186 at Cassandra.RowSet.IsExhausted() in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RowPopulators\RowSet.cs:line 80 at Cassandra.RowSet.<GetEnumerator>d__1.MoveNext() in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RowPopulators\RowSet.cs:line 145 {code} At the time of the client error there was an error logged by one of cassandra nodes in the cluster: {code} ERROR [ReadStage:24022] 2014-11-12 17:26:39,981 CassandraDaemon.java (line 217) Exception in thread Thread[ReadStage:24022,5,main] org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/ssd/cassandra/data/diadoc_letter_meta/metas_by_letter/diadoc_letter_meta-metas_by_letter-jb-3662-Data.db): corruption detected, chunk at 41506107 of length 31152. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:91) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:326) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85) at org.apache.cassandra.db.Column$1.computeNext(Column.java:75) at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:129) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:101) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:75) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1594) at org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1590) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1750) at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) at org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/ssd/cassandra/data/diadoc_letter_meta/metas_by_letter/diadoc_letter_meta-metas_by_letter-jb-3662-Data.db): corruption detected, chunk at 41506107 of length 31152. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:122) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87) ... 39 more Caused by: java.io.IOException: net.jpountz.lz4.LZ4Exception: Error decoding offset 19143 of input buffer at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:89) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:118) ... 40 more Caused by: net.jpountz.lz4.LZ4Exception: Error decoding offset 19143 of input buffer at net.jpountz.lz4.LZ4JNIFastDecompressor.decompress(LZ4JNIFastDecompressor.java:33) at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:84) ... 41 more {code} Client was performing read with CL=QUORUM and target data is stored with RF=3. Client was configured the following way: {code} cluster = Cluster.Builder() .AddContactPoints(settings.ContactEndPoints) .WithSocketOptions(new SocketOptions().SetTcpNoDelay(true)) .WithDefaultKeyspace(keyspaceName) .WithQueryTimeout(10000) // <- 10 sec client timeout .WithCompression(CompressionType.LZ4) .WithRetryPolicy(new DefaultRetryPolicy()) .WithLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy())) .WithReconnectionPolicy(new ExponentialReconnectionPolicy((long)TimeSpan.FromSeconds(1).TotalMilliseconds, (long)TimeSpan.FromMinutes(10).TotalMilliseconds)) .Build(); {code} Note that TokenAwarePolicy did not have any effect since we do not set routing keys on any statements yet. After we repaired that node using sstablescrub tool and nodetool repair command the error has gone. That is why I suspect that there is some interconnection between client timeout error in the driver and corrupted SSTable on server. Server Environment: Cassandra ReleaseVersion: 2.0.11 CentOS Linux release 7.0.1406 (Core) OpenJDK Runtime Environment (rhel-2.5.3.1.el7_0-x86_64 u71-b14) OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode) Client Environment: Windows Server 2012 R2 Standard x64 Cassandra C# driver built from git revision 930b8d04fbd7ac46a188ca2d596c55f6ed3a4318 (Fix keyspace race bug CSHARP-175)
    via by Andrew Kostousov,
  • Recently we encountered the following situation in production. Client tool performing reading data from some column family failed with System.TimeoutException at the time when there was no any significant load on the cluster. We were able to reproduce the error several times running this same tool on different machines at different times. Client error was: {code} INFO 2014-11-12 17:26:50,217 System.TimeoutException: The task didn't complete before timeout. at Cassandra.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout) in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\TaskHelper.cs:line 137 at Cassandra.RequestHandler`1.<FillRowSet>b__0(Byte[] pagingState) in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RequestHandler.cs:line 110 at Cassandra.RowSet.PageNext() in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RowPopulators\RowSet.cs:line 186 at Cassandra.RowSet.IsExhausted() in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RowPopulators\RowSet.cs:line 80 at Cassandra.RowSet.<GetEnumerator>d__1.MoveNext() in c:\projects\diadoc\dev\external\datastax-csharp-driver\src\Cassandra\RowPopulators\RowSet.cs:line 145 {code} At the time of the client error there was an error logged by one of cassandra nodes in the cluster: {code} ERROR [ReadStage:24022] 2014-11-12 17:26:39,981 CassandraDaemon.java (line 217) Exception in thread Thread[ReadStage:24022,5,main] org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/ssd/cassandra/data/diadoc_letter_meta/metas_by_letter/diadoc_letter_meta-metas_by_letter-jb-3662-Data.db): corruption detected, chunk at 41506107 of length 31152. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:91) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:326) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85) at org.apache.cassandra.db.Column$1.computeNext(Column.java:75) at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:129) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:101) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:75) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1594) at org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1590) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1750) at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) at org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/ssd/cassandra/data/diadoc_letter_meta/metas_by_letter/diadoc_letter_meta-metas_by_letter-jb-3662-Data.db): corruption detected, chunk at 41506107 of length 31152. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:122) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87) ... 39 more Caused by: java.io.IOException: net.jpountz.lz4.LZ4Exception: Error decoding offset 19143 of input buffer at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:89) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:118) ... 40 more Caused by: net.jpountz.lz4.LZ4Exception: Error decoding offset 19143 of input buffer at net.jpountz.lz4.LZ4JNIFastDecompressor.decompress(LZ4JNIFastDecompressor.java:33) at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:84) ... 41 more {code} Client was performing read with CL=QUORUM and target data is stored with RF=3. Client was configured the following way: {code} cluster = Cluster.Builder() .AddContactPoints(settings.ContactEndPoints) .WithSocketOptions(new SocketOptions().SetTcpNoDelay(true)) .WithDefaultKeyspace(keyspaceName) .WithQueryTimeout(10000) // <- 10 sec client timeout .WithCompression(CompressionType.LZ4) .WithRetryPolicy(new DefaultRetryPolicy()) .WithLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy())) .WithReconnectionPolicy(new ExponentialReconnectionPolicy((long)TimeSpan.FromSeconds(1).TotalMilliseconds, (long)TimeSpan.FromMinutes(10).TotalMilliseconds)) .Build(); {code} Note that TokenAwarePolicy did not have any effect since we do not set routing keys on any statements yet. After we repaired that node using sstablescrub tool and nodetool repair command the error has gone. That is why I suspect that there is some interconnection between client timeout error in the driver and corrupted SSTable on server. Server Environment: Cassandra ReleaseVersion: 2.0.11 CentOS Linux release 7.0.1406 (Core) OpenJDK Runtime Environment (rhel-2.5.3.1.el7_0-x86_64 u71-b14) OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode) Client Environment: Windows Server 2012 R2 Standard x64 Cassandra C# driver built from git revision 930b8d04fbd7ac46a188ca2d596c55f6ed3a4318 (Fix keyspace race bug CSHARP-175)
    via by Andrew Kostousov,
  • Spark job failing in YARN mode
    via Stack Overflow by Paul Trehiou
    ,
  • Fail to decompress a stream
    via GitHub by davies
    ,
  • groupBy/topN failure
    via by Federico Nieves,
    • org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/ssd/cassandra/data/diadoc_letter_meta/metas_by_letter/diadoc_letter_meta-metas_by_letter-jb-3662-Data.db): corruption detected, chunk at 41506107 of length 31152. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:91) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:326) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85) at org.apache.cassandra.db.Column$1.computeNext(Column.java:75) at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:129) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:101) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:75) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1594) at org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1590) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1750) at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) at org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/ssd/cassandra/data/diadoc_letter_meta/metas_by_letter/diadoc_letter_meta-metas_by_letter-jb-3662-Data.db): corruption detected, chunk at 41506107 of length 31152. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:122) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87) ... 39 more Caused by: java.io.IOException: net.jpountz.lz4.LZ4Exception: Error decoding offset 19143 of input buffer at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:89) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:118) ... 40 more Caused by: net.jpountz.lz4.LZ4Exception: Error decoding offset 19143 of input buffer at net.jpountz.lz4.LZ4JNIFastDecompressor.decompress(LZ4JNIFastDecompressor.java:33) at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:84) ... 41 more
    No Bugmate found.