java.lang.RuntimeException: java.lang.InterruptedException

Apache's JIRA Issue Tracker | Kai Wang | 11 months ago
  1. 0

    When I start the new node, I got this error: {noformat} ERROR [STREAM-IN-/192.168.0.10] 2015-12-22 15:37:56,302 StreamSession.java:524 - [Stream #bfc4e100-a8eb-11e5-bec5-67d8099a8b91] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(Unknown Source) ~[na:1.8.0_65] at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.8.0_65] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Unknown Source) [na:1.8.0_65] ERROR [Thread-22] 2015-12-22 15:37:56,302 CassandraDaemon.java:185 - Exception in thread Thread[Thread-22,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_65] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(Unknown Source) ~[na:1.8.0_65] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(Unknown Source) ~[na:1.8.0_65] at java.util.concurrent.ArrayBlockingQueue.put(Unknown Source) ~[na:1.8.0_65] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted INFO [STREAM-IN-/192.168.0.10] 2015-12-22 15:37:56,345 StreamResultFuture.java:182 - [Stream #bfc4e100-a8eb-11e5-bec5-67d8099a8b91] Session with /192.168.0.10 is complete WARN [STREAM-IN-/192.168.0.10] 2015-12-22 15:37:56,346 StreamResultFuture.java:209 - [Stream #bfc4e100-a8eb-11e5-bec5-67d8099a8b91] Stream failed ERROR [main] 2015-12-22 15:37:56,347 StorageService.java:1245 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.jar:na] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4] Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_65] WARN [main] 2015-12-22 15:37:56,353 StorageService.java:975 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS INFO [main] 2015-12-22 15:37:56,354 CassandraDaemon.java:596 - Waiting for gossip to settle before accepting client requests... INFO [main] 2015-12-22 15:38:04,355 CassandraDaemon.java:628 - No gossip backlog; proceeding INFO [main] 2015-12-22 15:38:04,411 Server.java:146 - Netty using native Epoll event loop INFO [main] 2015-12-22 15:38:04,442 Server.java:184 - Using Netty Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c, netty-codec=netty-codec-4.0.23.Final.208198c, netty-codec-http=netty-codec-http-4.0.23.Final.208198c, netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c, netty-common=netty-common-4.0.23.Final.208198c, netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c] INFO [main] 2015-12-22 15:38:04,443 Server.java:185 - Starting listening for CQL clients on /192.168.0.12:9042... INFO [main] 2015-12-22 15:38:04,495 ThriftServer.java:119 - Binding thrift service to /192.168.0.12:9160 INFO [Thread-23] 2015-12-22 15:38:04,501 ThriftServer.java:136 - Listening for thrift clients... ERROR [SharedPool-Worker-1] 2015-12-22 15:38:19,963 Message.java:611 - Unexpected exception during request; channel = [id: 0x03e85563, /192.168.0.10:55681 => /192.168.0.12:9042] java.lang.NullPointerException: null at org.apache.cassandra.auth.PasswordAuthenticator.doAuthenticate(PasswordAuthenticator.java:143) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:85) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.auth.PasswordAuthenticator.access$100(PasswordAuthenticator.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:181) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-2.2.4.jar:2.2.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_65] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Unknown Source) [na:1.8.0_65] {noformat} I tried to resume bootstrap via nodetool a couple of times. It made some progress initially (meaning some data was copied to the new node) but eventually all stopped with the same error.

    Apache's JIRA Issue Tracker | 11 months ago | Kai Wang
    java.lang.RuntimeException: java.lang.InterruptedException
  2. 0

    While running repair after upgrading to 2.2.2 I am getting many stream fail errors: {noformat} [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330 for range (59694553044959221,86389982480621619] failed with error [repair #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti vities, (59694553044959221,86389982480621619]] Sync failed between /10.240.81.104 and /10.240.134.221 (progress: 4%) {noformat} Logs from both sides of the stream: Sides 1 - {noformat} INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Creating new streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10 files(469491729 bytes) ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.81.104 is complete WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat} Side 2 - {noformat} INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for Repair INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061 StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Starting streaming to /10.240.134.221 INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063 StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Beginning stream session with /10.240.134.221 INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13 files(517391317 bytes) INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.134.221 is complete ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_60] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.2.jar:2.2.2] ... 11 common frames omitted WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat}

    Apache's JIRA Issue Tracker | 1 year ago | Omri Iluz
    java.lang.RuntimeException: java.lang.InterruptedException
  3. 0

    we got the same problem all the time when we add nodes to cluster. netstats: on HostA {noformat} /la-38395-big-Data.db 14792091851/14792091851 bytes(100%) sent to idx:0/HostB {noformat} on HostB {noformat} tmp-la-4-big-Data.db 2667087450/14792091851 bytes(18%) received from idx:0/HostA {noformat} After a while, Error on HostB {noformat} WARN [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:644 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Retrying for following error java.lang.IllegalArgumentException: Not enough bytes at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:173) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] ERROR [Thread-28] 2016-01-02 12:08:14,737 CassandraDaemon.java:185 - Exception in thread Thread[Thread-28,5,main] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66-internal] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[na:1.8.0_66-internal] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[na:1.8.0_66-internal] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 1 common frames omitted ERROR [STREAM-IN-/HostA] 2016-01-02 12:08:14,737 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred java.nio.channels.ClosedChannelException: null at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_66-internal] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_66-internal] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] INFO [STREAM-IN-/HostA] 2016-01-02 12:08:15,875 StreamResultFuture.java:182 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Session with /HostA is complete {noformat} Errors on HostA {noformat} WARN [STREAM-IN-/HostB] 2016-01-02 12:08:14,763 StreamResultFuture.java:209 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Stream failed ERROR [STREAM-OUT-/HostB] 2016-01-02 12:08:14,777 StreamSession.java:524 - [Stream #b91a4e90-b105-11e5-bd57-dd0cc3b4634c] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_66-internal] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_66-internal] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4] ... 11 common frames omitted {noformat}

    Apache's JIRA Issue Tracker | 11 months ago | xiaost
    java.lang.RuntimeException: java.lang.InterruptedException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    cluster - Cassandra nodetool repair failure - broken pipe - Server Fault

    serverfault.com | 5 months ago
    java.lang.RuntimeException: java.lang.InterruptedException
  6. 0

    While running repair after upgrading to 2.2.2 I am getting many stream fail errors: {noformat} [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330 for range (59694553044959221,86389982480621619] failed with error [repair #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti vities, (59694553044959221,86389982480621619]] Sync failed between /10.240.81.104 and /10.240.134.221 (progress: 4%) {noformat} Logs from both sides of the stream: Sides 1 - {noformat} INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Creating new streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063 StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Received streaming plan for Repair INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10 files(469491729 bytes) ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.81.104 is complete WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat} Side 2 - {noformat} INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for Repair INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061 StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Starting streaming to /10.240.134.221 INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063 StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550, ID#0] Beginning stream session with /10.240.134.221 INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098 StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550 ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13 files(517391317 bytes) INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Session with /10.240.134.221 is complete ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349 StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Streaming error occurred org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.2.jar:2.2.2] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.2.jar:2.2.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_60] at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_60] at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.2.jar:2.2.2] ... 11 common frames omitted WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349 StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Stream failed {noformat}

    Apache's JIRA Issue Tracker | 1 year ago | Omri Iluz
    java.lang.RuntimeException: java.lang.InterruptedException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.InterruptedException

      null

      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly()
    2. Java RT
      ArrayBlockingQueue.put
      1. java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(Unknown Source)[na:1.8.0_65]
      2. java.util.concurrent.locks.ReentrantLock.lockInterruptibly(Unknown Source)[na:1.8.0_65]
      3. java.util.concurrent.ArrayBlockingQueue.put(Unknown Source)[na:1.8.0_65]
      3 frames
    3. org.apache.cassandra
      CompressedInputStream$Reader.runMayThrow
      1. org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176)[apache-cassandra-2.2.4.jar:2.2.4]
      1 frame
    4. Apache Cassandra
      WrappedRunnable.run
      1. org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)[apache-cassandra-2.2.4.jar:2.2.4]
      1 frame
    5. Java RT
      Thread.run
      1. java.lang.Thread.run(Unknown Source)[na:1.8.0_65]
      1 frame