com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.

Couchbase JIRA | tal.maayani | 2 years ago
  1. 0

    We executed DHC client over the weekend and got some issues. Bottom line is that the DHC client disconnected and was not recovered after the weekend. See the errors below: [2015-05-15 00:06:54,012][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Error while refreshing bucket config, ignoring. com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [core-io-1.1.3.jar:1.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] [2015-05-15 00:14:25,846][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:14:41,239][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648881239 [2015-05-15 00:14:45,042][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:14:55,846][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:14:55,848][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:14:55,854][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:11,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648911240 [2015-05-15 00:15:15,043][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:15:25,850][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:41,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648941240

    Couchbase JIRA | 2 years ago | tal.maayani
    com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
  2. 0

    We executed DHC client over the weekend and got some issues. Bottom line is that the DHC client disconnected and was not recovered after the weekend. See the errors below: [2015-05-15 00:06:54,012][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Error while refreshing bucket config, ignoring. com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [core-io-1.1.3.jar:1.1.3] at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [core-io-1.1.3.jar:1.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] [2015-05-15 00:14:25,846][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:14:41,239][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648881239 [2015-05-15 00:14:45,042][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:14:55,846][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:14:55,847][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:14:55,848][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:14:55,854][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:11,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648911240 [2015-05-15 00:15:15,043][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:8092][ViewEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.kv.KeyValueHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: Identified Idle State, signalling config reload. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.endpoint.AbstractGenericHandler] [ip-172-31-27-164.ec2.internal/172.31.27.164:11210][KeyValueEndpoint]: KeepAlive fired [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] Received signal for outdated configuration. [2015-05-15 00:15:25,849][DEBUG][com.couchbase.client.core.config.ConfigurationProvider] New Bucket default config proposed. [2015-05-15 00:15:25,850][DEBUG][com.couchbase.client.core.config.refresher.CarrierRefresher] Completed refreshing config for bucket "default" [2015-05-15 00:15:41,240][DEBUG][org.eclipse.jetty.server.session] Scavenging sessions at 1431648941240

    Couchbase JIRA | 2 years ago | tal.maayani
    com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
  3. 0

    We use core io in our CBAS integration tests and frequently, we get failures due to request cancelled while on flight. The stack trace doesn't show any useful hint as to the cause behind that: java.lang.RuntimeException: java.lang.RuntimeException: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. at com.couchbase.analytics.test.dcp.CouchbaseConnectorStressTest.lambda$setUp$1(CouchbaseConnectorStressTest.java:97) Caused by: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:487) at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:468) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:616) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:868) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:834) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:826) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:43) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1276) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:185) at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:171) at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:811) at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:699) at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408) at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:402) at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745)

    Couchbase JIRA | 3 months ago | Abdullah Alamoudi
    java.lang.RuntimeException: java.lang.RuntimeException: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I'm working with Intel on MDS performance testing and here's how the cluster set up. 112: Data Service Only. 92: Index/Query service. (MOI, N1QL) Version: Couchbase Server 4.5 Beta. I started YCSB workloadE from client and here's are the error messages: com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying. at com.couchbase.client.core.retry.RetryHelper.retryOrCancel(RetryHelper.java:45) at com.couchbase.client.core.node.locate.QueryLocator.locateAndDispatch(QueryLocator.java:48) at com.couchbase.client.core.RequestHandler.dispatchRequest(RequestHandler.java:219) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:176) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:71) at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Matt and I looked into use strace and we found out there's SIGSEGV errors in the log. See attachment. Full logs are here: https://drive.google.com/drive/folders/0B1b0nb7ZnKdzM0V1eS1fVkYtVG8 I looked it up online and I see this: https://github.com/golang/go/issues/10984 This seems to be relate to Go runtime crash. Intel Env is behind firewall and I have access. Please let me know if you need access. Command to run from client side: strace -f -o workloade.out -e connect bin/ycsb run couchbase2 -P workloads/workloade -p couchbase.host=192.168.10.92 -threads 1 -p maxexecutiontime=120 -p operationcount=1000000 strace -f -o workloada.out -e connect bin/ycsb run couchbase2 -P workloads/workloada -p couchbase.host=192.168.10.112 -threads 4 -p maxexecutiontime=120 -p operationcount=1000000

    Couchbase JIRA | 7 months ago | Qi Zhu
    com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying.
  6. 0

    I'm working with Intel on MDS performance testing and here's how the cluster set up. 112: Data Service Only. 92: Index/Query service. (MOI, N1QL) Version: Couchbase Server 4.5 Beta. I started YCSB workloadE from client and here's are the error messages: com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying. at com.couchbase.client.core.retry.RetryHelper.retryOrCancel(RetryHelper.java:45) at com.couchbase.client.core.node.locate.QueryLocator.locateAndDispatch(QueryLocator.java:48) at com.couchbase.client.core.RequestHandler.dispatchRequest(RequestHandler.java:219) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:176) at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:71) at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Matt and I looked into use strace and we found out there's SIGSEGV errors in the log. See attachment. Full logs are here: https://drive.google.com/drive/folders/0B1b0nb7ZnKdzM0V1eS1fVkYtVG8 I looked it up online and I see this: https://github.com/golang/go/issues/10984 This seems to be relate to Go runtime crash. Intel Env is behind firewall and I have access. Please let me know if you need access. Command to run from client side: strace -f -o workloade.out -e connect bin/ycsb run couchbase2 -P workloads/workloade -p couchbase.host=192.168.10.92 -threads 1 -p maxexecutiontime=120 -p operationcount=1000000 strace -f -o workloada.out -e connect bin/ycsb run couchbase2 -P workloads/workloada -p couchbase.host=192.168.10.112 -threads 4 -p maxexecutiontime=120 -p operationcount=1000000

    Couchbase JIRA | 7 months ago | Qi Zhu
    com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying.

    Root Cause Analysis

    1. com.couchbase.client.core.RequestCancelledException

      Request cancelled in-flight.

      at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations()
    2. com.couchbase.client
      DefaultThreadFactory$DefaultRunnableDecorator.run
      1. com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281)[core-io-1.1.3.jar:1.1.3]
      2. com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262)[core-io-1.1.3.jar:1.1.3]
      3. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527)[core-io-1.1.3.jar:1.1.3]
      4. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521)[core-io-1.1.3.jar:1.1.3]
      5. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351)[core-io-1.1.3.jar:1.1.3]
      6. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798)[core-io-1.1.3.jar:1.1.3]
      7. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767)[core-io-1.1.3.jar:1.1.3]
      8. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759)[core-io-1.1.3.jar:1.1.3]
      9. com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743)[core-io-1.1.3.jar:1.1.3]
      10. com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615)[core-io-1.1.3.jar:1.1.3]
      11. com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)[core-io-1.1.3.jar:1.1.3]
      12. com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)[core-io-1.1.3.jar:1.1.3]
      13. com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[core-io-1.1.3.jar:1.1.3]
      14. com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[core-io-1.1.3.jar:1.1.3]
      14 frames
    3. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)[na:1.8.0_40]
      1 frame