java.nio.channels.ClosedChannelException: null

Apache's JIRA Issue Tracker | xiaost | 11 months ago
  1. 0

    we got the same problem all the time when we add nodes to cluster. netstats: on HostA {noformat} /la-38395-big-Data.db 14792091851/14792091851 bytes(100%) sent to idx:0/HostB {noformat} on HostB {noformat} tmp-la-4-big-Data.db 2667087450/14792091851 bytes(18%) received from idx:0/HostA {noformat} After a while, Error on HostB {noformat} WARN [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:644 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Retrying for following error java.lang.IllegalArgumentException: Not enough bytes at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:173) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] ERROR [Thread-28] 2016-01-02 12:08:14,737 CassandraDaemon.java:185 - Exception in thread Thread[Thread-28,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-internal] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_66-internal] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_66-internal] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_66-internal] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] INFO [STREAM-IN-/HostA] 2016-01-02 12:08:15,875 StreamResultFuture.java:182 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Session with /HostA is complete {noformat} Errors on HostA {noformat} WARN [STREAM-IN-/HostB] 2016-01-02 12:08:14,763 StreamResultFuture.java:209 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Stream failed ERROR [STREAM-OUT-/HostB] 2016-01-02 12:08:14,777 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66-internal] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat}

    Apache's JIRA Issue Tracker | 11 months ago | xiaost
    java.nio.channels.ClosedChannelException: null
  2. 0

    cluster - Cassandra nodetool repair failure - broken pipe - Server Fault

    serverfault.com | 5 months ago
    java.nio.channels.ClosedChannelException: null
  3. 0

    we got the same problem all the time when we add nodes to cluster. netstats: on HostA {noformat} /la-38395-big-Data.db 14792091851/14792091851 bytes(100%) sent to idx:0/HostB {noformat} on HostB {noformat} tmp-la-4-big-Data.db 2667087450/14792091851 bytes(18%) received from idx:0/HostA {noformat} After a while, Error on HostB {noformat} WARN [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:644 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Retrying for following error java.lang.IllegalArgumentException: Not enough bytes at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:173) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] ERROR [Thread-28] 2016-01-02 12:08:14,737 CassandraDaemon.java:185 - Exception in thread Thread[Thread-28,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-internal] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_66-internal] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_66-internal] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_66-internal] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] INFO [STREAM-IN-/HostA] 2016-01-02 12:08:15,875 StreamResultFuture.java:182 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Session with /HostA is complete {noformat} Errors on HostA {noformat} WARN [STREAM-IN-/HostB] 2016-01-02 12:08:14,763 StreamResultFuture.java:209 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Stream failed ERROR [STREAM-OUT-/HostB] 2016-01-02 12:08:14,777 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66-internal] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat}

    Apache's JIRA Issue Tracker | 11 months ago | xiaost
    java.nio.channels.ClosedChannelException: null
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassandra nodetool repair failure - broken pipe

    Server Fault | 11 months ago | Greg M.
    java.nio.channels.ClosedChannelException: null
  6. 0

    hello I got 2 nodes in 2 DC. Each node own 100% data of keyspace hugespace. Keyspace have 21 tables with 2TB data Biggest table have 1.6 TB of data. Biggest sstable 1,3 TB. Schemats: {noformat} KEYSPACE hugespace WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '1'}; CREATE TABLE hugespace.content ( y int, m int, d int, ts bigint, ha text, co text, he text, ids bigint, ifr text, js text, PRIMARY KEY ((y, m, d), ts, ha) ) WITH CLUSTERING ORDER BY (ts ASC, ha ASC) AND bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; CREATE INDEX content_ids_idx ON hugespace.content (ids); {noformat} I tried to add one node (target 6 node in DC1) to DC1. Names: Existing node in DC1 = nodeDC1 Existing node in DC2 = nodeDC2 New node joining DC1 = joiningDC1 joiningDC1 {noformat} INFO [main] 2016-01-04 12:17:55,535 StorageService.java:1176 - JOINING: Starting to bootstrap... INFO [main] 2016-01-04 12:17:55,802 StreamResultFuture.java:86 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Executing streaming plan for Bootstrap INFO [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,803 StreamSession.java:232 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Starting streaming to /nodeDC1 INFO [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,803 StreamSession.java:232 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Starting streaming to /nodeDC2 DEBUG [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,803 ConnectionHandler.java:82 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for incoming stream DEBUG [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,803 ConnectionHandler.java:82 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for incoming stream DEBUG [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,806 ConnectionHandler.java:87 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for outgoing stream DEBUG [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,806 ConnectionHandler.java:87 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending stream init for outgoing stream DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:17:55,810 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (5 requests, 0 files} DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:55,810 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (2 requests, 0 files} INFO [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,810 StreamCoordinator.java:213 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Beginning stream session with /nodeDC2 INFO [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,810 StreamCoordinator.java:213 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Beginning stream session with /nodeDC1 DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,821 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (0 requests, 1 files} INFO [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,822 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 1 files(161 bytes), sending 0 files(0 bytes) DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,828 CompressedStreamReader.java:67 - reading file from /nodeDC2, repairedAt = 1451483586917 DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,831 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId: 5bc52802-de25-35ed-aeab-188eecebb090, #0, version: la, format: BIG, estimated keys: 128, transfer size: 161, compressed?: true, repairedAt: 1451483586917, level: 0), file: /cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/tmp-la-1-big-Data.db) DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:55,831 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Received (5bc52802-de25-35ed-aeab-188eecebb090, #0) DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,832 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Complete DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:56,049 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Complete DEBUG [StreamReceiveTask:1] 2016-01-04 12:17:56,049 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /nodeDC2 INFO [StreamReceiveTask:1] 2016-01-04 12:17:56,049 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /nodeDC2 is complete DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,848 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (0 requests, 194 files} INFO [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,849 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 194 files(2130405704708 bytes), sending 0 files(0 bytes) DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,882 CompressedStreamReader.java:67 - reading file from /nodeDC1, repairedAt = 1451476116723 DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:01,418 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId: 99e93a30-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 128, transfer size: 56333243, compressed?: true, repairedAt: 1451476116723, level: 0), file: /cassandra/data/hugespace/other_data-99e93a30814a11e5a573a94b3a1b0e16/tmp-la-1-big-Data.db) DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:01,419 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Received (99e93a30-814a-11e5-a573-a94b3a1b0e16, #0) DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:01,596 CompressedStreamReader.java:67 - reading file from /nodeDC1, repairedAt = 0 DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,371 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId: a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2, version: la, format: BIG, estimated keys: 128, transfer size: 1693486699, compressed?: true, repairedAt: 0, level: 0), file: /cassandra/data/hugespace/content_demo-a5f465809e5e11e59e43a94b3a1b0e16/tmp-la-1-big-Data.db) DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:55,374 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Received (a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2) ERROR [Thread-4] 2016-01-04 12:18:55,375 CassandraDaemon.java:185 - Exception in thread Thread[Thread-4,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,375 StreamSession.java:524 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_65] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_65] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:55,376 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Session Failed DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,488 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /nodeDC1 INFO [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,489 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /nodeDC1 is complete WARN [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,489 StreamResultFuture.java:209 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Stream failed ERROR [main] 2016-01-04 12:18:55,490 StorageService.java:1245 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.jar:na] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4] Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] WARN [main] 2016-01-04 12:18:55,496 StorageService.java:975 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS {noformat} nodeDC1 {noformat} DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:56,848 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (0 requests, 194 files} INFO [STREAM-IN-/joiningDC1] 2016-01-04 12:17:56,848 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 194 files 2130405704708 bytes) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:56,851 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending File (Header (cfId: 99e93a30-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 128, transfer size: 56333243, compressed?: true, repairedAt: 1451476116723, level: 0), file: /cassandra/data/hugespace/other_data-99e93a30814a11e5a573a94b3a1b0e16/la-202-big-Data.db) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:59,596 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending File (Header (cfId: a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2, version: la, format: BIG, estimated keys: 128, transfer size: 1693486699, compressed?: true, repairedAt: 0, level: 0), file: /cassandra/data/hugespace/content_demo-a5f465809e5e11e59e43a94b3a1b0e16/la-1280-big-Data.db) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:01,422 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Received (99e93a30-814a-11e5-a573-a94b3a1b0e16, #0) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,375 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Received (a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,920 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Session Failed DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,920 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /joiningDC1 INFO [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,921 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /joiningDC1 is complete WARN [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,923 StreamResultFuture.java:209 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Stream failed ERROR [STREAM-OUT-/joiningDC1] 2016-01-04 12:18:55,925 StreamSession.java:524 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat} nodeDC2 _other timezone +1_ no error {noformat} INFO [STREAM-INIT-/joiningDC1:46377] 2016-01-04 13:17:55,807 StreamResultFuture.java:111 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Creating new streaming plan for Bootstrap INFO [STREAM-INIT-/joiningDC1:46377] 2016-01-04 13:17:55,808 StreamResultFuture.java:118 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Received streaming plan for Bootstrap INFO [STREAM-INIT-/joiningDC1:46380] 2016-01-04 13:17:55,809 StreamResultFuture.java:118 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4, ID#0] Received streaming plan for Bootstrap DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,819 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (2 requests, 0 files} DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,819 StreamSession.java:347 - ViewFilter for 1/1 sstables DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,820 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Prepare (0 requests, 1 files} INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,820 StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4 ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(161 bytes) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,820 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending File (Header (cfId: 5bc52802-de25-35ed-aeab-188eecebb090, #0, version: la, format: BIG, estimated keys: 128, transfer size: 161, compressed?: true, repairedAt: 1451483586917, level: 0), file: /srv/cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/la-1-big-Data.db) DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,831 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Received (5bc52802-de25-35ed-aeab-188eecebb090, #0) DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,832 ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Sending Complete DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,049 ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Complete DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050 ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection handler on /joiningDC1 INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050 StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Session with /joiningDC1 is complete INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050 StreamResultFuture.java:214 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] All sessions completed {noformat} I also ran sstableverify two times on nodeDC1 and i get this error (2 times for the same sstables) {noformat} ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@799ed08e) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@546942821:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-12252-big-Data.db was not released before the reference was garbage collected ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2c747998) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@218678140:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13487-big-Data.db was not released before the reference was garbage collected ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@63a80e20) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@652932560:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13499-big-Data.db was not released before the reference was garbage collected ERROR 23:13:50 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@295df02c) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup@1644545368:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13497-big-Data.db was not released before the reference was garbage collected {noformat} sstable size 176 GB, 94GB, 3,6GB, 3,6GB all in the same table as biggest one - 1,3 TB I albo ran nodetool repair on nodeDC1 and i get the same error nodeDC1 {noformat} ERROR [Thread-10] 2015-12-31 05:09:27,960 CassandraDaemon.java:185 - Exception in thread Thread[Thread-10,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted DEBUG [STREAM-IN-/nodeDC2] 2015-12-31 05:09:27,960 ConnectionHandler.java:266 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Received File (Header (cfId: 94be9870-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 1, transfer size: 5190031760, compressed?: true, repairedAt: 1451517863052, level: 0), file: /cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/tmp-la-13501-big-Data.db) ERROR [STREAM-IN-/nodeDC2] 2015-12-31 05:09:27,967 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_65] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_65] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] DEBUG [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,235 ConnectionHandler.java:110 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Closing stream connection handler on /nodeDC2 INFO [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,235 StreamResultFuture.java:182 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Session with /nodeDC2 is complete WARN [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,236 StreamResultFuture.java:209 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Stream failed ERROR [STREAM-OUT-/nodeDC2] 2015-12-31 05:09:34,236 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted WARN [RepairJobTask:17] 2015-12-31 05:09:34,236 RepairJob.java:162 - [repair #75cfdc00-af4c-11e5-8429-a94b3a1b0e16] content sync failed ERROR [RepairJobTask:17] 2015-12-31 05:09:34,238 RepairSession.java:290 - [repair #75cfdc00-af4c-11e5-8429-a94b3a1b0e16] Session completed with the following error org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] ERROR [RepairJobTask:17] 2015-12-31 05:09:34,239 RepairRunnable.java:243 - Repair session 75cfdc00-af4c-11e5-8429-a94b3a1b0e16 for range (-308435053110486424,-206175645991596694] failed with error Stream failed org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] {noformat} nodeDC2 _timezone +1_ {noformat} ERROR [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,590 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted DEBUG [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591 ConnectionHandler.java:110 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Closing stream connection handler on /nodeDC1 INFO [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591 StreamResultFuture.java:182 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Session with /nodeDC1 is complete WARN [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591 StreamResultFuture.java:209 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Stream failed ERROR [Thread-25] 2015-12-31 06:09:33,932 CassandraDaemon.java:185 - Exception in thread Thread[Thread-25,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted DEBUG [STREAM-IN-/nodeDC1] 2015-12-31 06:09:33,931 ConnectionHandler.java:266 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Received File (Header (cfId: 94be9870-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated keys: 1, transfer size: 5344434128, compressed?: true, repairedAt: 1451517863052, level: 0), file: /srv/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/tmp-la-13949-big-Data.db) ERROR [STREAM-IN-/nodeDC1] 2015-12-31 06:09:33,935 StreamSession.java:524 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred java.lang.RuntimeException: Outgoing stream handler has been closed at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:132) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:587) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:476) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:267) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] {noformat} Please advice.

    Apache's JIRA Issue Tracker | 11 months ago | Jack Money
    java.nio.channels.ClosedChannelException: null

  1. davidvanlaatum 6 times, last 6 months ago
4 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.nio.channels.ClosedChannelException

    null

    at sun.nio.ch.SocketChannelImpl.ensureReadOpen()
  2. Java RT
    SocketChannelImpl.read
    1. sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)[na:1.8.0_66-internal]
    2. sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)[na:1.8.0_66-internal]
    2 frames
  3. org.apache.cassandra
    ConnectionHandler$IncomingMessageHandler.run
    1. org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)[apache-cassandra-2.2.4.jar:2.2.4]
    2. org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)[apache-cassandra-2.2.4.jar:2.2.4]
    2 frames
  4. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:745)[na:1.8.0_66-internal]
    1 frame