com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.

Couchbase JIRA | tal.maayani | 2 years ago
  1. 0

    We executed DHC client over the weekend and got some issues. Bottom line is that the DHC client disconnected and was not recovered after the weekend. See the errors below: [2015-05-15 00:06:54,012][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Error while refreshing bucket config, ignoring. com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [core-io-1.1.3.jar:1.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] [2015-05-15 00:14:25,846][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:14:41,239][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648881239 [2015-05-15 00:14:45,042][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:14:55,846][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:14:55,848][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:14:55,854][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:11,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648911240 [2015-05-15 00:15:15,043][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:15:25,850][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:41,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648941240

    Couchbase JIRA | 2 years ago | tal.maayani
    com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
  2. 0

    We executed DHC client over the weekend and got some issues. Bottom line is that the DHC client disconnected and was not recovered after the weekend. See the errors below: [2015-05-15 00:06:54,012][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Error while refreshing bucket config, ignoring. com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [core-io-1.1.3.jar:1.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] [2015-05-15 00:14:25,846][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:14:41,239][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648881239 [2015-05-15 00:14:45,042][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:14:55,846][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:14:55,848][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:14:55,854][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:11,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648911240 [2015-05-15 00:15:15,043][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:15:25,850][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:41,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648941240

    Couchbase JIRA | 2 years ago | tal.maayani
    com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
  3. 0

    I'm working with Intel on MDS performance testing and here's how the cluster set up. 112: Data Service Only. 92: Index/Query service. (MOI, N1QL) Version: Couchbase Server 4.5 Beta. I started YCSB workloadE from client and here's are the error messages: com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying. at com.couchbase.client.core.retry.RetryHelper.retryOrCancel(RetryHelper.java:45) at com.couchbase.client.core.node.locate.QueryLocator.locateAndDispatch(QueryLocator.java:48) at com.couchbase.client.core.RequestHandler.dispatchRequest(RequestHandler.java:219) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:176) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:71) at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Matt and I looked into use strace and we found out there's SIGSEGV errors in the log. See attachment. Full logs are here: https://drive.google.com/drive/folders/0B1b0nb7ZnKdzM0V1eS1fVkYtVG8 I looked it up online and I see this: https://github.com/golang/go/issues/10984 This seems to be relate to Go runtime crash. Intel Env is behind firewall and I have access. Please let me know if you need access. Command to run from client side: strace -f -o workloade.out -e connect bin/ycsb run couchbase2 -P workloads/workloade -p couchbase.host=192.168.10.92 -threads 1 -p maxexecutiontime=120 -p operationcount=1000000 strace -f -o workloada.out -e connect bin/ycsb run couchbase2 -P workloads/workloada -p couchbase.host=192.168.10.112 -threads 4 -p maxexecutiontime=120 -p operationcount=1000000

    Couchbase JIRA | 6 months ago | Qi Zhu
    com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I'm working with Intel on MDS performance testing and here's how the cluster set up. 112: Data Service Only. 92: Index/Query service. (MOI, N1QL) Version: Couchbase Server 4.5 Beta. I started YCSB workloadE from client and here's are the error messages: com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying. at com.couchbase.client.core.retry.RetryHelper.retryOrCancel(RetryHelper.java:45) at com.couchbase.client.core.node.locate.QueryLocator.locateAndDispatch(QueryLocator.java:48) at com.couchbase.client.core.RequestHandler.dispatchRequest(RequestHandler.java:219) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:176) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:71) at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Matt and I looked into use strace and we found out there's SIGSEGV errors in the log. See attachment. Full logs are here: https://drive.google.com/drive/folders/0B1b0nb7ZnKdzM0V1eS1fVkYtVG8 I looked it up online and I see this: https://github.com/golang/go/issues/10984 This seems to be relate to Go runtime crash. Intel Env is behind firewall and I have access. Please let me know if you need access. Command to run from client side: strace -f -o workloade.out -e connect bin/ycsb run couchbase2 -P workloads/workloade -p couchbase.host=192.168.10.92 -threads 1 -p maxexecutiontime=120 -p operationcount=1000000 strace -f -o workloada.out -e connect bin/ycsb run couchbase2 -P workloads/workloada -p couchbase.host=192.168.10.112 -threads 4 -p maxexecutiontime=120 -p operationcount=1000000

    Couchbase JIRA | 6 months ago | Qi Zhu
    com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying.

    Root Cause Analysis

    1. com.couchbase.client.core.RequestCancelledException

      Request cancelled in-flight.

      at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations()
    2. com.couchbase.client
      DefaultThreadFactory$DefaultRunnableDecorator.run
      1. com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281)[core-io-1.1.3.jar:1.1.3]
      2. com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262)[core-io-1.1.3.jar:1.1.3]
      3. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527)[core-io-1.1.3.jar:1.1.3]
      4. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521)[core-io-1.1.3.jar:1.1.3]
      5. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351)[core-io-1.1.3.jar:1.1.3]
      6. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798)[core-io-1.1.3.jar:1.1.3]
      7. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767)[core-io-1.1.3.jar:1.1.3]
      8. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759)[core-io-1.1.3.jar:1.1.3]
      9. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743)[core-io-1.1.3.jar:1.1.3]
      10. com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615)[core-io-1.1.3.jar:1.1.3]
      11. com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)[core-io-1.1.3.jar:1.1.3]
      12. com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)[core-io-1.1.3.jar:1.1.3]
      13. com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[core-io-1.1.3.jar:1.1.3]
      14. com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[core-io-1.1.3.jar:1.1.3]
      14 frames
    3. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)[na:1.8.0_40]
      1 frame