com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket

Couchbase JIRA | Michael Blow | 2 months ago
  1. 0

    CouchbaseConnectorStressTest.testConnectDisconnectCycle performs the following in a loop: {noformat} Client client = Client.configure() .eventLoopGroup(ioPool()) .bucket(bucket) .password(cbPassword) .hostnames(cbNodes) .connectionNameGenerator(() -> threadName) .controlParam(DcpControl.Names.CONNECTION_BUFFER_SIZE, 16384) .bufferAckWatermark(80) .build(); client.controlEventHandler(ReferenceCountUtil::release); client.dataEventHandler(event -> { try { client.acknowledgeBuffer(event); } finally { ReferenceCountUtil.release(event); } }); timedAwait("client.connect()", client.connect()); timedAwait("client.disconnect()", client.disconnect()); {noformat} Between 10-20 seconds of iterations, I observe many exceptions like the following on the console: {noformat} ## testConnectDisconnectCycle START ### Starting 4 threads with run time of 60 seconds. #### [thread 0] Starting thread (will trace every 10 seconds) #### [thread 1] Starting thread (will trace every 10 seconds) #### [thread 2] Starting thread (will trace every 10 seconds) #### [thread 3] Starting thread (will trace every 10 seconds) #### [thread 1] ...iteration 1338... #### [thread 0] ...iteration 1001... #### [thread 3] ...iteration 213... #### [thread 2] ...iteration 450... [cb-io-1-6] 19:58:26 WARN DefaultChannelPipeline:299 - An exception 'java.lang.IllegalStateException: complete already: DefaultChannelPromise@752e0392(failure: java.io.IOException: Connection reset by peer)' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at com.couchbase.client.deps.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) at com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399) at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745) {noformat} The error does not get thrown to the client, and iterations continue. Eventually after 30-40 seconds (and 1000s of iterations), I do get an error out of all threads: {noformat} #### [thread 3] ...iteration 1801... #### [thread 2] Failing fast due to 1 failure(s): [com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket] #### [thread 2] Exiting thread after 2320 iterations. #### [thread 1] Failing fast due to 1 failure(s): [com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket] #### [thread 1] Exiting thread after 2872 iterations. #### [thread 3] Failing fast due to 1 failure(s): [com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket] #### [thread 3] Exiting thread after 1808 iterations. ### Completed 8792 iterations with 4 failure(s). ## testConnectDisconnectCycle FAILED (com.couchbase.client.dcp.error.BootstrapException) com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket at com.couchbase.client.dcp.Client$4.call(Client.java:290) at com.couchbase.client.dcp.Client$4.call(Client.java:287) at rx.Completable$24$1.onError(Completable.java:1735) at rx.internal.operators.CompletableOnSubscribeConcatArray$ConcatInnerSubscriber.onError(CompletableOnSubscribeConcatArray.java:63) at com.couchbase.client.dcp.conductor.HttpStreamingConfigProvider$4$1.operationComplete(HttpStreamingConfigProvider.java:194) at com.couchbase.client.dcp.conductor.HttpStreamingConfigProvider$4$1.operationComplete(HttpStreamingConfigProvider.java:177) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:120) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87) at com.couchbase.client.dcp.transport.netty.ConnectInterceptingHandler.exceptionCaught(ConnectInterceptingHandler.java:108) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:290) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:269) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:261) at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:423) at com.couchbase.client.deps.io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79) at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:144) at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:230) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:290) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:269) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:261) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1261) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:290) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:269) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:899) at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87) at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399) at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at com.couchbase.client.deps.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) at com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) ... 7 more {noformat}

    Couchbase JIRA | 2 months ago | Michael Blow
    com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket
  2. 0

    Not sure if this is because of StevesAddons but it has only started after it was added

    GitHub | 2 years ago | AoKMiKeY
    java.io.IOException: Connection reset by peer
  3. 0

    TcpServer: LEAK: ByteBuf.release() was not called in before it's garbage-collected

    GitHub | 2 years ago | riteshmodi
    java.io.IOException: Connection reset by peer
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    LEAK: ByteBuf.release() was not called in before it's garbage-collected. Spring Reactor TcpServer

    Stack Overflow | 2 years ago
    java.io.IOException: Connection reset by peer
  6. 0

    Closed Channel Exception with valid p12

    GitHub | 1 year ago | tonicebrian
    java.io.IOException: Connection reset by peer

  1. Raggyman 2 times, last 2 months ago
  2. bpbhat77 4 times, last 3 months ago
  3. kid 28 times, last 2 months ago
  4. Piz 4 times, last 4 months ago
  5. poroszd 1 times, last 5 months ago
7 more registered users
88 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.IOException

    Connection reset by peer

    at sun.nio.ch.FileDispatcherImpl.read0()
  2. Java RT
    SocketChannelImpl.read
    1. sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    2. sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    3. sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    4. sun.nio.ch.IOUtil.read(IOUtil.java:192)
    5. sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    5 frames
  3. com.couchbase.client
    DefaultThreadFactory$DefaultRunnableDecorator.run
    1. com.couchbase.client.deps.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
    2. com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
    3. com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
    4. com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
    5. com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572)
    6. com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513)
    7. com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427)
    8. com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399)
    9. com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
    10. com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    10 frames
  4. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:745)
    1 frame