org.apache.cassandra.streaming.StreamException: Stream failed

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • sstableloader: Stream failed
    via by Ralf Steppacher,
  • Re: sstableloader: Stream failed
    via by Unknown author,
  • Cassandra sstableloader - Connection refused
    via Stack Overflow by riccamini
    ,
  • Cassandra nodetool decommission
    via Stack Overflow by Vinod Jayachandran
    ,
  • we got the same problem all the time when we add nodes to cluster. netstats: on HostA {noformat} /la-38395-big-Data.db 14792091851/14792091851 bytes(100%) sent to idx:0/HostB {noformat} on HostB {noformat} tmp-la-4-big-Data.db 2667087450/14792091851 bytes(18%) received from idx:0/HostA {noformat} After a while, Error on HostB {noformat} WARN [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:644 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Retrying for following error java.lang.IllegalArgumentException: Not enough bytes at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:173) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] ERROR [Thread-28] 2016-01-02 12:08:14,737 CassandraDaemon.java:185 - Exception in thread Thread[Thread-28,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-internal] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_66-internal] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_66-internal] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_66-internal] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] INFO [STREAM-IN-/HostA] 2016-01-02 12:08:15,875 StreamResultFuture.java:182 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Session with /HostA is complete {noformat} Errors on HostA {noformat} WARN [STREAM-IN-/HostB] 2016-01-02 12:08:14,763 StreamResultFuture.java:209 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Stream failed ERROR [STREAM-OUT-/HostB] 2016-01-02 12:08:14,777 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66-internal] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat}
    via by xiaost,
  • When attempting to bootstrap a new node into a 2.1.3 cluster, the stream source fails with a {{NullPointerException}}: {noformat} ERROR [STREAM-IN-/10.xx.x.xxx] 2015-06-13 00:02:01,264 StreamSession.java:477 - [Stream #60e8c120- 115f-11e5-9fee-xxxxxxxxxxxx] Streaming error occurred java.lang.NullPointerException: null at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:1277) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.streaming.StreamSession.getSSTableSectionsForRanges(StreamSession.java:313) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.streaming.StreamSession.addTransferRanges(StreamSession.java:266) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:493) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:425) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:251) ~[apache-cassandra-2.1.3.jar:2.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] INFO [STREAM-IN-/10.xx.x.xxx] 2015-06-13 00:02:01,265 StreamResultFuture.java:180 - [Stream #60e8c120-115f-11e5-9fee-xxxxxxxxxxxx] Session with /10.xx.x.xx1 is complete {noformat} _Update (2015-06-26):_ I can also reproduce this on 2.1.7, though without the NPE on the stream-from side. Stream source / existing node: {noformat} INFO [STREAM-IN-/10.64.32.178] 2015-06-26 06:48:53,060 StreamResultFuture.java:180 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9] Session with /10.64.32.178 is complete INFO [STREAM-IN-/10.64.32.178] 2015-06-26 06:48:53,064 StreamResultFuture.java:212 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9] All sessions completed {noformat} Stream sink / bootstrapping node: {noformat} INFO [StreamReceiveTask:57] 2015-06-26 06:48:53,061 StreamResultFuture.java:180 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9] Session with /10.64.32.160 is complete WARN [StreamReceiveTask:57] 2015-06-26 06:48:53,062 StreamResultFuture.java:207 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9] Stream failed INFO [CompactionExecutor:2885] 2015-06-26 06:48:53,062 ColumnFamilyStore.java:906 - Enqueuing flush of compactions_in_progress: 428 (0%) on-heap, 379 (0%) off-heap INFO [MemtableFlushWriter:959] 2015-06-26 06:48:53,063 Memtable.java:346 - Writing Memtable-compactions_in_progress@1203013482(294 serialized bytes, 12 ops, 0%/0% of on/off-heap limit) ERROR [main] 2015-06-26 06:48:53,063 CassandraDaemon.java:541 - Exception encountered during startup java.lang.RuntimeException: Error during boostrap: Stream failed at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1137) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:927) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:723) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:605) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.7.jar:2.1.7] Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.1.7.jar:2.1.7] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:409) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:682) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:645) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:139) ~[apache-cassandra-2.1.7.jar:2.1.7] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79] {noformat}
    via by Eric Evans,
  • Cassandra avoid JOINING state go to RUNNING
    via Stack Overflow by koblas
    ,
    • org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:115) Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:241) at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    3 more bugmates