java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed

Apache's JIRA Issue Tracker | xiaost | 11 months ago
  1. 0

    we got the same problem all the time when we add nodes to cluster. netstats: on HostA {noformat} /la-38395-big-Data.db 14792091851/14792091851 bytes(100%) sent to idx:0/HostB {noformat} on HostB {noformat} tmp-la-4-big-Data.db 2667087450/14792091851 bytes(18%) received from idx:0/HostA {noformat} After a while, Error on HostB {noformat} WARN [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:644 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Retrying for following error java.lang.IllegalArgumentException: Not enough bytes at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:173) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] ERROR [Thread-28] 2016-01-02 12:08:14,737 CassandraDaemon.java:185 - Exception in thread Thread[Thread-28,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-internal] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_66-internal] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_66-internal] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_66-internal] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] INFO [STREAM-IN-/HostA] 2016-01-02 12:08:15,875 StreamResultFuture.java:182 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Session with /HostA is complete {noformat} Errors on HostA {noformat} WARN [STREAM-IN-/HostB] 2016-01-02 12:08:14,763 StreamResultFuture.java:209 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Stream failed ERROR [STREAM-OUT-/HostB] 2016-01-02 12:08:14,777 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66-internal] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat}

    Apache's JIRA Issue Tracker | 11 months ago | xiaost
    java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
  2. 0

    we got the same problem all the time when we add nodes to cluster. netstats: on HostA {noformat} /la-38395-big-Data.db 14792091851/14792091851 bytes(100%) sent to idx:0/HostB {noformat} on HostB {noformat} tmp-la-4-big-Data.db 2667087450/14792091851 bytes(18%) received from idx:0/HostA {noformat} After a while, Error on HostB {noformat} WARN [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:644 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Retrying for following error java.lang.IllegalArgumentException: Not enough bytes at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:173) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] ERROR [Thread-28] 2016-01-02 12:08:14,737 CassandraDaemon.java:185 - Exception in thread Thread[Thread-28,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-internal] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_66-internal] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_66-internal] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_66-internal] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] INFO [STREAM-IN-/HostA] 2016-01-02 12:08:15,875 StreamResultFuture.java:182 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Session with /HostA is complete {noformat} Errors on HostA {noformat} WARN [STREAM-IN-/HostB] 2016-01-02 12:08:14,763 StreamResultFuture.java:209 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Stream failed ERROR [STREAM-OUT-/HostB] 2016-01-02 12:08:14,777 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66-internal] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat}

    Apache's JIRA Issue Tracker | 11 months ago | xiaost
    java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
  3. 0

    hello I got 2 nodes in 2 DC. Each node own 100% data of keyspace hugespace. Keyspace have 21 tables with 2TB data Biggest table have 1.6 TB of data. Biggest sstable 1,3 TB. Schemats: {noformat} KEYSPACE hugespace WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '1'}; CREATE TABLE hugespace.content ( y int, m int, d int, ts bigint, ha text, co text, he text, ids bigint, ifr text, js text, PRIMARY KEY ((y, m, d), ts, ha) ) WITH CLUSTERING ORDER BY (ts ASC, ha ASC) AND bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; CREATE INDEX content_ids_idx ON hugespace.content (ids); {noformat} I tried to add one node (target 6 node in DC1) to DC1. Names: Existing node in DC1 = nodeDC1 Existing node in DC2 = nodeDC2 New node joining DC1 = joiningDC1 joiningDC1 {noformat} INFO [main] 2016-01-04 12:17:55,535 StorageService.java:1176 - JOINING: Starting to bootstrap... INFO [main] 2016-01-04 12:17:55,802 StreamResultFuture.java:86 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Executing streaming plan for Bootstrap INFO [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,803 StreamSession.java:232 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Starting streaming to /nodeDC1 INFO [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,803 StreamSession.java:232 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Starting streaming to /nodeDC2 DEBUG [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,803 ConnectionHandler.java:82 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for incoming stream DEBUG [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,803 ConnectionHandler.java:82 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for incoming stream DEBUG [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,806 ConnectionHandler.java:87 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for outgoing stream DEBUG [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,806 ConnectionHandler.java:87 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for outgoing stream DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:17:55,810 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (5 requests, 0 files} DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:55,810 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (2 requests, 0 files} INFO [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,810 StreamCoordinator.java:213 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Beginning stream session with /nodeDC2 INFO [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,810 StreamCoordinator.java:213 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Beginning stream session with /nodeDC1 DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,821 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (0 requests, 1 files} INFO [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,822 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 1 files(161 bytes), sending 0 files(0 bytes) DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,828 CompressedStreamReader.java:67 - reading file from /nodeDC2, repairedAt = 1451483586917 DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,831 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId: 5bc52802-de25-35ed-aeab-188eecebb090, #0, version: la, format: BIG, estimated keys: 128, transfer size: 161, compressed?: true, repairedAt: 1451483586917, level: 0), file: /cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/tmp-la-1-big-Data.db) DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:55,831 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Received (5bc52802-de25-35ed-aeab-188eecebb090, #0) DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,832 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Complete DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:56,049 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Complete DEBUG [StreamReceiveTask:1] 2016-01-04 12:17:56,049 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /nodeDC2 INFO [StreamReceiveTask:1] 2016-01-04 12:17:56,049 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /nodeDC2 is complete DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,848 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (0 requests, 194 files} INFO [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,849 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 194 files(2130405704708 bytes), sending 0 files(0 bytes) DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,882 CompressedStreamReader.java:67 - reading file from /nodeDC1, repairedAt = 1451476116723 DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:01,418 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId: 99e93a30-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 128, transfer size: 56333243, compressed?: true, repairedAt: 1451476116723, level: 0), file: /cassandra/data/hugespace/other_data-99e93a30814a11e5a573a94b3a1b0e16/tmp-la-1-big-Data.db) DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:01,419 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Received (99e93a30-814a-11e5-a573-a94b3a1b0e16, #0) DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:01,596 CompressedStreamReader.java:67 - reading file from /nodeDC1, repairedAt = 0 DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,371 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId: a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2, version: la, format: BIG, estimated keys: 128, transfer size: 1693486699, compressed?: true, repairedAt: 0, level: 0), file: /cassandra/data/hugespace/content_demo-a5f465809e5e11e59e43a94b3a1b0e16/tmp-la-1-big-Data.db) DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:55,374 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Received (a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2) ERROR [Thread-4] 2016-01-04 12:18:55,375 CassandraDaemon.java:185 - Exception in thread Thread[Thread-4,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,375 StreamSession.java:524 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_65] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_65] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:55,376 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Session Failed DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,488 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /nodeDC1 INFO [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,489 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /nodeDC1 is complete WARN [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,489 StreamResultFuture.java:209 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Stream failed ERROR [main] 2016-01-04 12:18:55,490 StorageService.java:1245 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.jar:na] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4] Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] WARN [main] 2016-01-04 12:18:55,496 StorageService.java:975 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS {noformat} nodeDC1 {noformat} DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:56,848 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (0 requests, 194 files} INFO [STREAM-IN-/joiningDC1] 2016-01-04 12:17:56,848 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 194 files 2130405704708 bytes) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:56,851 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending File (Header (cfId: 99e93a30-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 128, transfer size: 56333243, compressed?: true, repairedAt: 1451476116723, level: 0), file: /cassandra/data/hugespace/other_data-99e93a30814a11e5a573a94b3a1b0e16/la-202-big-Data.db) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:59,596 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending File (Header (cfId: a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2, version: la, format: BIG, estimated keys: 128, transfer size: 1693486699, compressed?: true, repairedAt: 0, level: 0), file: /cassandra/data/hugespace/content_demo-a5f465809e5e11e59e43a94b3a1b0e16/la-1280-big-Data.db) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:01,422 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Received (99e93a30-814a-11e5-a573-a94b3a1b0e16, #0) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,375 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Received (a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,920 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Session Failed DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,920 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /joiningDC1 INFO [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,921 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /joiningDC1 is complete WARN [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,923 StreamResultFuture.java:209 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Stream failed ERROR [STREAM-OUT-/joiningDC1] 2016-01-04 12:18:55,925 StreamSession.java:524 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat} nodeDC2 _other timezone +1_ no error {noformat} INFO [STREAM-INIT-/joiningDC1:46377] 2016-01-04 13:17:55,807 StreamResultFuture.java:111 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Creating new streaming plan for Bootstrap INFO [STREAM-INIT-/joiningDC1:46377] 2016-01-04 13:17:55,808 StreamResultFuture.java:118 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Received streaming plan for Bootstrap INFO [STREAM-INIT-/joiningDC1:46380] 2016-01-04 13:17:55,809 StreamResultFuture.java:118 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Received streaming plan for Bootstrap DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,819 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (2 requests, 0 files} DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,819 StreamSession.java:347 - ViewFilter for 1/1 sstables DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,820 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (0 requests, 1 files} INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,820 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(161 bytes) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,820 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending File (Header (cfId: 5bc52802-de25-35ed-aeab-188eecebb090, #0, version: la, format: BIG, estimated keys: 128, transfer size: 161, compressed?: true, repairedAt: 1451483586917, level: 0), file: /srv/cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/la-1-big-Data.db) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,831 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Received (5bc52802-de25-35ed-aeab-188eecebb090, #0) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,832 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Complete DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,049 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Complete DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /joiningDC1 INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /joiningDC1 is complete INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050 StreamResultFuture.java:214 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] All sessions completed {noformat} I also ran sstableverify two times on nodeDC1 and i get this error (2 times for the same sstables) {noformat} ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@799ed08e) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@546942821:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-12252-big-Data.db was not released before the reference was garbage collected ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2c747998) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@218678140:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13487-big-Data.db was not released before the reference was garbage collected ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@63a80e20) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@652932560:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13499-big-Data.db was not released before the reference was garbage collected ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@295df02c) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@1644545368:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13497-big-Data.db was not released before the reference was garbage collected {noformat} sstable size 176 GB, 94GB, 3,6GB, 3,6GB all in the same table as biggest one - 1,3 TB I albo ran nodetool repair on nodeDC1 and i get the same error nodeDC1 {noformat} ERROR [Thread-10] 2015-12-31 05:09:27,960 CassandraDaemon.java:185 - Exception in thread Thread[Thread-10,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted DEBUG [STREAM-IN-/nodeDC2] 2015-12-31 05:09:27,960 ConnectionHandler.java:266 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Received File (Header (cfId: 94be9870-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 1, transfer size: 5190031760, compressed?: true, repairedAt: 1451517863052, level: 0), file: /cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/tmp-la-13501-big-Data.db) ERROR [STREAM-IN-/nodeDC2] 2015-12-31 05:09:27,967 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_65] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_65] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] DEBUG [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,235 ConnectionHandler.java:110 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Closing stream connection handler on /nodeDC2 INFO [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,235 StreamResultFuture.java:182 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Session with /nodeDC2 is complete WARN [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,236 StreamResultFuture.java:209 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Stream failed ERROR [STREAM-OUT-/nodeDC2] 2015-12-31 05:09:34,236 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted WARN [RepairJobTask:17] 2015-12-31 05:09:34,236 RepairJob.java:162 - [repair #75cfdc00-af4c-11e5-8429-a94b3a1b0e16] content sync failed ERROR [RepairJobTask:17] 2015-12-31 05:09:34,238 RepairSession.java:290 - [repair #75cfdc00-af4c-11e5-8429-a94b3a1b0e16] Session completed with the following error org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] ERROR [RepairJobTask:17] 2015-12-31 05:09:34,239 RepairRunnable.java:243 - Repair session 75cfdc00-af4c-11e5-8429-a94b3a1b0e16 for range (-308435053110486424,-206175645991596694] failed with error Stream failed org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] {noformat} nodeDC2 _timezone +1_ {noformat} ERROR [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,590 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted DEBUG [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591 ConnectionHandler.java:110 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Closing stream connection handler on /nodeDC1 INFO [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591 StreamResultFuture.java:182 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Session with /nodeDC1 is complete WARN [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591 StreamResultFuture.java:209 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Stream failed ERROR [Thread-25] 2015-12-31 06:09:33,932 CassandraDaemon.java:185 - Exception in thread Thread[Thread-25,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted DEBUG [STREAM-IN-/nodeDC1] 2015-12-31 06:09:33,931 ConnectionHandler.java:266 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Received File (Header (cfId: 94be9870-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 1, transfer size: 5344434128, compressed?: true, repairedAt: 1451517863052, level: 0), file: /srv/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/tmp-la-13949-big-Data.db) ERROR [STREAM-IN-/nodeDC1] 2015-12-31 06:09:33,935 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred java.lang.RuntimeException: Outgoing stream handler has been closed at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:132) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:587) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:476) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:267) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] {noformat} Please advice.

    Apache's JIRA Issue Tracker | 11 months ago | Jack Money
    java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Using SSTableSimpleUnsortedWriter with all-key-columns table

    Stack Overflow | 1 year ago | Errandir
    java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
  6. 0

    Cassandra....bootstrapping : Edward Capriolo

    edwardcapriolo.com | 3 months ago
    java.lang.RuntimeException: Error during boostrap: Stream failed

    7 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.cassandra.streaming.StreamException

      Stream failed

      at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure()
    2. org.apache.cassandra
      StreamEventJMXNotifier.onFailure
      1. org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)[apache-cassandra-2.2.4.jar:2.2.4]
      1 frame
    3. Guava
      AbstractFuture.setException
      1. com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)[guava-16.0.jar:na]
      2. com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)[guava-16.0.jar:na]
      3. com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[guava-16.0.jar:na]
      4. com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[guava-16.0.jar:na]
      5. com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[guava-16.0.jar:na]
      5 frames
    4. org.apache.cassandra
      ConnectionHandler$IncomingMessageHandler.run
      1. org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)[apache-cassandra-2.2.4.jar:2.2.4]
      2. org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)[apache-cassandra-2.2.4.jar:2.2.4]
      3. org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434)[apache-cassandra-2.2.4.jar:2.2.4]
      4. org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529)[apache-cassandra-2.2.4.jar:2.2.4]
      5. org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)[apache-cassandra-2.2.4.jar:2.2.4]
      5 frames
    5. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)[na:1.8.0_65]
      1 frame