java.io.IOException: CF 64aecb30-11f7-11e6-89d2-9d1dd801d7e2 was dropped during streaming

Apache's JIRA Issue Tracker | Anastasia Osintseva | 10 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    I have in cluster 2 DC, in each DC - 2 Nodes. I wanted to add 1 node to each DC. One node has been added successfully after I had made scrubing. Now I'm trying to add node to another DC, but get error: org.apache.cassandra.streaming.StreamException: Stream failed. After scrubing and repair I get the same error. {noformat} ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 Keyspace.java:492 - Unknown exception caught while attempting to update MaterializedView! messages_dump.messages java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large for the maxiumum size of 33554432 at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169) [apache-cassandra-3.0.5.jar:3.0.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamReceiveTask.java:214 - Error applying streamed data: java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large for the maxiumum size of 33554432 at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169) ~[apache-cassandra-3.0.5.jar:3.0.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamSession.java:520 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Streaming error occurred java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large for the maxiumum size of 33554432 at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169) ~[apache-cassandra-3.0.5.jar:3.0.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] DEBUG [StreamReceiveTask:5] 2016-04-27 00:33:21,082 ConnectionHandler.java:110 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Closing stream connection handler on /88.9.99.92 DEBUG [STREAM-OUT-/88.9.99.92] 2016-04-27 00:33:21,082 ConnectionHandler.java:341 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Sending Session Failed INFO [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamResultFuture.java:182 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Session with /88.9.99.92 is complete WARN [StreamReceiveTask:5] 2016-04-27 00:33:21,182 StreamResultFuture.java:209 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Stream failed ERROR [main] 2016-04-27 00:33:21,259 StorageService.java:1300 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1295) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:745) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:610) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:333) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679) [apache-cassandra-3.0.5.jar:3.0.5] Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.5.jar:3.0.5] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:525) ~[apache-cassandra-3.0.5.jar:3.0.5] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:216) ~[apache-cassandra-3.0.5.jar:3.0.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_11] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_11] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_11] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_11] {noformat} I set commitlog_segment_size_in_mb: 128, but it didn't help.

    Apache's JIRA Issue Tracker | 10 months ago | Anastasia Osintseva
    java.io.IOException: CF 64aecb30-11f7-11e6-89d2-9d1dd801d7e2 was dropped during streaming
  2. 0

    [CASSANDRA-8387] Schema inconsistency (cached vs schema_columnfamilies) - ASF JIRA

    apache.org | 1 year ago
    java.io.IOException: CF 5bc7c980-74b5-11e4-9131-d9b94a3d8927 was dropped during streaming
  3. 0

    Streaming fails

    GitHub | 2 years ago | elad
    java.io.IOException: CF b9dda069-4d0a-37c3-b0e8-8c4cf88a4a9f was dropped during streaming
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassandra failure with more than one Cassandra node

    GitHub | 2 months ago | plwhite
    java.io.IOException: CF 205243d0-bb0d-11e6-8fc5-e9b99718e783 was dropped during streaming
  6. 0

    While running some tests on a 3-node cluster running C* 2.1.1 we encountered a problem creating the same table schema twice(on different nodes). One thing to note is that one of the nodes clock was ~4 seconds behind the others, but I don't think that's the problem since the exception was reproduced here aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg39560.html. While running the same create table statement more than once(on different clients) the logs outputted this on one of the nodes: {noformat} (node x.x.x.1): 2014-11-25T16:11:44.651+0100 INFO [SharedPool-Worker-2] MigrationManager.java:248 Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@45c290de[cfId=5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... 2014-11-25T16:11:44.667+0100 INFO [MigrationStage:1] DefsTables.java:373 Loading org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted 2014-11-25T16:11:44.859+0100 ERROR [SharedPool-Worker-2] ErrorMessage.java:243 Unexpected exception during request java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4- b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:374) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:249) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:114) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:80) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:481) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:133) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.1.jar:2.1.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.1.jar:2.1.1] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d ; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_51] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 17 common frames omitted Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_51] ... 1 common frames omitted Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted {noformat} In my understanding this means that it first created the table(with UUID 5e..) and then got a schema update from another node with UUID 5b.. . And the node who created the 5b.. table had this: {noformat} (node x.x.x.2): 2014-11-25T16:11:40.580+0100 INFO [SharedPool-Worker-1] MigrationManager.java:248 Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@225c1975[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted {noformat} And the last node had this: {noformat} (node x.x.x.3): 2014-11-25T16:11:44.363+0100 INFO [MigrationStage:1] DefsTables.java:373 Loading org.apache.cassandra.config.CFMetaData@767512bf[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted {noformat} Not a problem at all, since they all seemed to agree that 5b.. was the schema to go with and everything went fine with inserting data into the table. The data directory contained the 5b.. directory and not 5e.. (on all nodes). The problem occured later when node x.x.x.3 was restarted it created the directory test/test-5e.. in the data directory. When performing a repair on the node it gave these kinds of exceptions: {noformat} (node x.x.x.3): 2014-11-25T16:21:31.726+0100 WARN [STREAM-IN-/x.x.x.2] StreamSession.java:590 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] Retrying for following error java.io.IOException: CF 5bc7c980-74b5-11e4-9131-d9b94a3d8927 was dropped during streaming at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:71) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) [apache-cassandra-2.1.1.jar:2.1.1] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] ... 2014-11-25T16:21:31.726+0100 ERROR [STREAM-IN-/x.x.x.2] StreamSession.java:470 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:89) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] ... 2014-11-25T16:21:31.738+0100 ERROR [AntiEntropySessions:1] RepairSession.java:303 [repair #bbf34630-74b6-11e4-baf6-fbb25862655d] session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #bbf34630-74b6-11e4-baf6-fbb25862655d on test/test, (3708140006327119811,3721360916800446611]] Sync failed between /x.x.x.3 and /x.x.x.2 at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:389) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:126) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] {noformat} And the other nodes had exceptions like: {noformat} (node x.x.x.2) 2014-11-25T16:21:28.145+0100 ERROR [STREAM-OUT-/x.x.x.3] StreamSession.java:470 [Stream #bc309d50-74b6-11e4-baf6-fbb25862655d] Streaming error occurred java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_51] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_51] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_51] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_51] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_51] at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[na:1.7.0_51] at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.7.0_51] at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.7.0_51] at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.7.0_51] at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.7.0_51] at java.nio.channels.Channels$1.write(Channels.java:155) ~[na:1.7.0_51] at org.apache.cassandra.io.util.DataOutputStreamPlus.write(DataOutputStreamPlus.java:45) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.io.util.AbstractDataOutput.writeLong(AbstractDataOutput.java:232) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:406) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:402) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:60) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:47) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:135) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:120) ~[apache-cassandra-2.1.1.jar:2.1.1] {noformat} In System.schema_columnfamilies the id for test.test is 5e.. on all nodes, but the nodes that hadn't restarted still used the 5b.. id for the table (atleast in the data directory, probably a cached id). The problem seems to be that the schema change for 5e.. is rejected on all nodes (java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException thrown), while the mutation for it still gets applied in schema_columnfamilies.

    Apache's JIRA Issue Tracker | 2 years ago | Marcus Olsson
    java.io.IOException: CF 5bc7c980-74b5-11e4-9131-d9b94a3d8927 was dropped during streaming

    Root Cause Analysis

    1. java.io.IOException

      CF 64aecb30-11f7-11e6-89d2-9d1dd801d7e2 was dropped during streaming

      at org.apache.cassandra.streaming.compress.CompressedStreamReader.read()
    2. org.apache.cassandra
      ConnectionHandler$IncomingMessageHandler.run
      1. org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:76)[apache-cassandra-3.0.5.jar:3.0.5]
      2. org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)[apache-cassandra-3.0.5.jar:3.0.5]
      3. org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)[apache-cassandra-3.0.5.jar:3.0.5]
      4. org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)[apache-cassandra-3.0.5.jar:3.0.5]
      5. org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)[apache-cassandra-3.0.5.jar:3.0.5]
      5 frames
    3. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)[na:1.8.0_11]
      1 frame