com.stumbleupon.async.CallbackOverflowError: Too many callbacks in Deferred@811087221(state=PENDING, result=null, callback=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@3467b7a6 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@71653f4e -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@315c24d8 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@405e4c1b -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@498d0931 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@114ac001 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@41e04c14 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@73eceea7 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@5289a56a -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@15043df2 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@56c753aa -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@424a1603 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@5d7749d5 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@69a81424 ....(size=16383) when attempting to add cb=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@31ad1009@833425417, eb=passthrough@922150289

GitHub | bstoll | 5 months ago
  1. 0

    CallbackOverflowError with many tag values

    GitHub | 5 months ago | bstoll
    com.stumbleupon.async.CallbackOverflowError: Too many callbacks in Deferred@811087221(state=PENDING, result=null, callback=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@3467b7a6 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@71653f4e -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@315c24d8 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@405e4c1b -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@498d0931 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@114ac001 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@41e04c14 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@73eceea7 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@5289a56a -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@15043df2 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@56c753aa -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@424a1603 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@5d7749d5 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@69a81424 ....(size=16383) when attempting to add cb=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@31ad1009@833425417, eb=passthrough@922150289
  2. 0

    Calling flush() too often results in a CallbackOverflowError

    GitHub | 2 years ago | tsuna
    com.stumbleupon.async.CallbackOverflowError: Too many callbacks in Deferred@1750061096(state=PENDING, result=null, callback=multi-action response -> notify Def erredGroup@1340811160 -> notify DeferredGroup@1467258149 -> [...]

    Root Cause Analysis

    1. com.stumbleupon.async.CallbackOverflowError

      Too many callbacks in Deferred@811087221(state=PENDING, result=null, callback=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@3467b7a6 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@71653f4e -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@315c24d8 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@405e4c1b -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@498d0931 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@114ac001 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@41e04c14 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@73eceea7 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@5289a56a -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@15043df2 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@56c753aa -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@424a1603 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@5d7749d5 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@69a81424 ....(size=16383) when attempting to add cb=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@31ad1009@833425417, eb=passthrough@922150289

      at com.stumbleupon.async.Deferred.addCallbacks()
    2. SUAsync Library
      Deferred.addCallback
      1. com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:669)[async-1.4.0.jar:na]
      2. com.stumbleupon.async.Deferred.addCallback(Deferred.java:724)[async-1.4.0.jar:na]
      2 frames
    3. net.opentsdb.tsd
      QueryRpc$1QueriesCB.call
      1. net.opentsdb.tsd.HttpJsonSerializer.formatQueryAsyncV1(HttpJsonSerializer.java:874)[tsdb-2.3.0-RC1.jar:]
      2. net.opentsdb.tsd.QueryRpc$1QueriesCB.call(QueryRpc.java:260)[tsdb-2.3.0-RC1.jar:]
      3. net.opentsdb.tsd.QueryRpc$1QueriesCB.call(QueryRpc.java:231)[tsdb-2.3.0-RC1.jar:]
      3 frames
    4. SUAsync Library
      Deferred.callback
      1. com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)[async-1.4.0.jar:na]
      2. com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)[async-1.4.0.jar:na]
      3. com.stumbleupon.async.Deferred.callback(Deferred.java:1005)[async-1.4.0.jar:na]
      4. com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:173)[async-1.4.0.jar:na]
      5. com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158)[async-1.4.0.jar:na]
      6. com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36)[async-1.4.0.jar:na]
      7. com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97)[async-1.4.0.jar:na]
      8. com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)[async-1.4.0.jar:na]
      9. com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)[async-1.4.0.jar:na]
      10. com.stumbleupon.async.Deferred.callback(Deferred.java:1005)[async-1.4.0.jar:na]
      10 frames
    5. net.opentsdb.core
      TsdbQuery$1ScannerCB.call
      1. net.opentsdb.core.TsdbQuery$1ScannerCB.close(TsdbQuery.java:873)[tsdb-2.3.0-RC1.jar:]
      2. net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:633)[tsdb-2.3.0-RC1.jar:]
      3. net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:575)[tsdb-2.3.0-RC1.jar:]
      3 frames
    6. SUAsync Library
      Deferred.callback
      1. com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)[async-1.4.0.jar:na]
      2. com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)[async-1.4.0.jar:na]
      3. com.stumbleupon.async.Deferred.access$300(Deferred.java:430)[async-1.4.0.jar:na]
      4. com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)[async-1.4.0.jar:na]
      5. com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)[async-1.4.0.jar:na]
      6. com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)[async-1.4.0.jar:na]
      7. com.stumbleupon.async.Deferred.callback(Deferred.java:1005)[async-1.4.0.jar:na]
      7 frames
    7. org.hbase.async
      RegionClient.decode
      1. org.hbase.async.HBaseRpc.callback(HBaseRpc.java:698)[asynchbase-1.7.1.jar:na]
      2. org.hbase.async.RegionClient.decode(RegionClient.java:1516)[asynchbase-1.7.1.jar:na]
      3. org.hbase.async.RegionClient.decode(RegionClient.java:88)[asynchbase-1.7.1.jar:na]
      3 frames
    8. Netty
      SimpleChannelUpstreamHandler.handleUpstream
      1. org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)[netty-3.9.4.Final.jar:na]
      2. org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)[netty-3.9.4.Final.jar:na]
      3. org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[netty-3.9.4.Final.jar:na]
      3 frames
    9. org.hbase.async
      RegionClient.handleUpstream
      1. org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1206)[asynchbase-1.7.1.jar:na]
      1 frame
    10. Netty
      DefaultChannelPipeline.sendUpstream
      1. org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[netty-3.9.4.Final.jar:na]
      2. org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[netty-3.9.4.Final.jar:na]
      3. org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)[netty-3.9.4.Final.jar:na]
      4. org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)[netty-3.9.4.Final.jar:na]
      5. org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36)[netty-3.9.4.Final.jar:na]
      6. org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[netty-3.9.4.Final.jar:na]
      7. org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[netty-3.9.4.Final.jar:na]
      8. org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294)[netty-3.9.4.Final.jar:na]
      9. org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[netty-3.9.4.Final.jar:na]
      10. org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[netty-3.9.4.Final.jar:na]
      11. org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)[netty-3.9.4.Final.jar:na]
      11 frames
    11. org.hbase.async
      HBaseClient$RegionClientPipeline.sendUpstream
      1. org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3108)[asynchbase-1.7.1.jar:na]
      1 frame
    12. Netty
      DeadLockProofWorker$1.run
      1. org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)[netty-3.9.4.Final.jar:na]
      2. org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)[netty-3.9.4.Final.jar:na]
      3. org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)[netty-3.9.4.Final.jar:na]
      4. org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)[netty-3.9.4.Final.jar:na]
      5. org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)[netty-3.9.4.Final.jar:na]
      6. org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)[netty-3.9.4.Final.jar:na]
      7. org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)[netty-3.9.4.Final.jar:na]
      8. org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)[netty-3.9.4.Final.jar:na]
      9. org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)[netty-3.9.4.Final.jar:na]
      9 frames
    13. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)[na:1.8.0_91]
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)[na:1.8.0_91]
      3. java.lang.Thread.run(Unknown Source)[na:1.8.0_91]
      3 frames