java.lang.IllegalArgumentException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • We have 30 nodes in one DC, 25 nodes in another. We are running 2.0.1. Two nodes are joining the ring, but one of them failed ARN [STREAM-IN-/10.4.197.53] 2014-01-09 19:41:40,418 StreamResultFuture.java (line 209) [Stream #e515d6e0-795d-11e3-b74a-b72892248056] Stream failed ERROR [main] 2014-01-09 19:41:40,418 CassandraDaemon.java (line 459) Exception encountered during startup java.lang.RuntimeException: Error during boostrap: Stream failed at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:901) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:670) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:529) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:428) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185) at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:321) at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:501) at org.apache.cassandra.streaming.StreamSession.messageReceived(Stre amSession.java:376) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293) at java.lang.Thread.run(Thread.java:744) ERROR [StorageServiceShutdownHook] 2014-01-09 19:41:40,428 CassandraDaemon.java (line 185) Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:312) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:361) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:96) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:494) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:744)
    via by Shao-Chuan Wang,
  • Cassandra staled bootstrap
    via Stack Overflow by Ara Sadoyan
    ,
  • Streaming fails
    via GitHub by elad
    ,
  • While running repair after upgrading to 2.2.2 I am getting many stream fail errors: {noformat} [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330 for range (59694553044959221,86389982480621619] failed with error [repair #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti vities, (59694553044959221,86389982480621619]] Sync failed between /10.240.81.104 and /10.240.134.221 (progress: 4%) {noformat} Logs from both sides of the stream: Sides 1 - {noformat} INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Creating new streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10 files(469491729 bytes) ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.81.104 is complete WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat} Side 2 - {noformat} INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for Repair INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061 StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Starting streaming to /10.240.134.221 INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063 StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Beginning stream session with /10.240.134.221 INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13 files(517391317 bytes) INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.134.221 is complete ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_60] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.2.jar:2.2.2] ... 11 common frames omitted WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat}
    via by Omri Iluz,
  • While running repair after upgrading to 2.2.2 I am getting many stream fail errors: {noformat} [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330 for range (59694553044959221,86389982480621619] failed with error [repair #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti vities, (59694553044959221,86389982480621619]] Sync failed between /10.240.81.104 and /10.240.134.221 (progress: 4%) {noformat} Logs from both sides of the stream: Sides 1 - {noformat} INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Creating new streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10 files(469491729 bytes) ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.81.104 is complete WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat} Side 2 - {noformat} INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for Repair INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061 StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Starting streaming to /10.240.134.221 INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063 StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Beginning stream session with /10.240.134.221 INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13 files(517391317 bytes) INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.134.221 is complete ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_60] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.2.jar:2.2.2] ... 11 common frames omitted WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat}
    via by Omri Iluz,
  • Cassandra : data not replicated on new node
    via Stack Overflow by Bertrand
    ,
  • While running some tests on a 3-node cluster running C* 2.1.1 we encountered a problem creating the same table schema twice(on different nodes). One thing to note is that one of the nodes clock was ~4 seconds behind the others, but I don't think that's the problem since the exception was reproduced here aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg39560.html. While running the same create table statement more than once(on different clients) the logs outputted this on one of the nodes: {noformat} (node x.x.x.1): 2014-11-25T16:11:44.651+0100 INFO [SharedPool-Worker-2] MigrationManager.java:248 Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@45c290de[cfId=5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... 2014-11-25T16:11:44.667+0100 INFO [MigrationStage:1] DefsTables.java:373 Loading org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted 2014-11-25T16:11:44.859+0100 ERROR [SharedPool-Worker-2] ErrorMessage.java:243 Unexpected exception during request java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4- b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:374) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:249) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:114) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:80) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:481) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:133) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.1.jar:2.1.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.1.jar:2.1.1] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d ; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_51] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 17 common frames omitted Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_51] ... 1 common frames omitted Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted {noformat} In my understanding this means that it first created the table(with UUID 5e..) and then got a schema update from another node with UUID 5b.. . And the node who created the 5b.. table had this: {noformat} (node x.x.x.2): 2014-11-25T16:11:40.580+0100 INFO [SharedPool-Worker-1] MigrationManager.java:248 Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@225c1975[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted {noformat} And the last node had this: {noformat} (node x.x.x.3): 2014-11-25T16:11:44.363+0100 INFO [MigrationStage:1] DefsTables.java:373 Loading org.apache.cassandra.config.CFMetaData@767512bf[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] ... java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) at org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) ~[apache-cassandra-2.1.1.jar:2.1.1] ... 11 common frames omitted {noformat} Not a problem at all, since they all seemed to agree that 5b.. was the schema to go with and everything went fine with inserting data into the table. The data directory contained the 5b.. directory and not 5e.. (on all nodes). The problem occured later when node x.x.x.3 was restarted it created the directory test/test-5e.. in the data directory. When performing a repair on the node it gave these kinds of exceptions: {noformat} (node x.x.x.3): 2014-11-25T16:21:31.726+0100 WARN [STREAM-IN-/x.x.x.2] StreamSession.java:590 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] Retrying for following error java.io.IOException: CF 5bc7c980-74b5-11e4-9131-d9b94a3d8927 was dropped during streaming at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:71) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) [apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) [apache-cassandra-2.1.1.jar:2.1.1] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] ... 2014-11-25T16:21:31.726+0100 ERROR [STREAM-IN-/x.x.x.2] StreamSession.java:470 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:89) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] ... 2014-11-25T16:21:31.738+0100 ERROR [AntiEntropySessions:1] RepairSession.java:303 [repair #bbf34630-74b6-11e4-baf6-fbb25862655d] session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #bbf34630-74b6-11e4-baf6-fbb25862655d on test/test, (3708140006327119811,3721360916800446611]] Sync failed between /x.x.x.3 and /x.x.x.2 at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:389) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:126) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.1.jar:2.1.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] {noformat} And the other nodes had exceptions like: {noformat} (node x.x.x.2) 2014-11-25T16:21:28.145+0100 ERROR [STREAM-OUT-/x.x.x.3] StreamSession.java:470 [Stream #bc309d50-74b6-11e4-baf6-fbb25862655d] Streaming error occurred java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_51] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_51] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_51] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_51] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_51] at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[na:1.7.0_51] at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.7.0_51] at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.7.0_51] at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.7.0_51] at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.7.0_51] at java.nio.channels.Channels$1.write(Channels.java:155) ~[na:1.7.0_51] at org.apache.cassandra.io.util.DataOutputStreamPlus.write(DataOutputStreamPlus.java:45) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.io.util.AbstractDataOutput.writeLong(AbstractDataOutput.java:232) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:406) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:402) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:60) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:47) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:135) ~[apache-cassandra-2.1.1.jar:2.1.1] at org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:120) ~[apache-cassandra-2.1.1.jar:2.1.1] {noformat} In System.schema_columnfamilies the id for test.test is 5e.. on all nodes, but the nodes that hadn't restarted still used the 5b.. id for the table (atleast in the data directory, probably a cached id). The problem seems to be that the schema change for 5e.. is rejected on all nodes (java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException thrown), while the mutation for it still gets applied in schema_columnfamilies.
    via by Marcus Olsson,
    • java.lang.IllegalArgumentException: Unknown type -128 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:89) at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:283) at java.lang.Thread.run(Thread.java:744)

    Users with the same issue

    Unknown visitor
    Unknown visitor1 times, last one,