com.datastax.driver.core.exceptions.ProtocolError: An unexpected error occurred on host /10.10.11.121:9042. This is a bug in library, please report: Invalid value for the paging state

DataStax JIRA | Jakub Janecek | 11 months ago
  1. 0

    We have encountered the following exception during scan of our Cassandra cluster: {code} 13:19:03.903 ERROR c.a.u.d.LoggingLatencyTracker$ [cassandra-nio-worker-4]: Exception when making query to /10.10.11.121:9042, latency was 7 ms (LoggingLatencyTracker.scala:9) com.datastax.driver.core.exceptions.ProtocolError: An unexpected error occurred on host /10.10.11.121:9042. This is a bug in library, please report: Invalid value for the paging state at com.datastax.driver.core.Responses$Error.asException(Responses.java:92) [cassandra-driver-core-2.1.9.jar:na] at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:433) [cassandra-driver-core-2.1.9.jar:na] at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1009) [cassandra-driver-core-2.1.9.jar:na] at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:932) [cassandra-driver-core-2.1.9.jar:na] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) [netty-handler-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] {code} Because the scan is taking several weeks we need it to be restartable. Therefore we are using {{resultSet.getExecutionInfo.getPagingStateUnsafe}} to retrieve the current paging state and store it in a file. If the process is restarted it will use this paging state to continue from where it stopped before via {{Statement.setPagingStateUnsafe(state)}}. We are running the scan on several nodes at once to speed it up so the query looks like this: {code} select key, column, value, token(key) from keyspace.cf where token(key) >= 0 and token(key) < 100 {code} The process was running fine for several days until it failed on one of the nodes with this exception. The reason is unknown and we would like you to help us debug it (please, let me know how we can help you) and in the end possibly fix a bug in the driver. There is another issue we encountered. All of our scans (as I mentioned we run several parallel scans at once on different token ranges to speed things up) finished "successfully" at certain point. We our computing the progress based on the current token (we know the start and the end so we can compute how far we are) and all the scans finished at around 30%. They refused to continue even after restart and the only option was to adjust the token range to continue. We know for sure there are more data to be scanned because using CQLSH we can list them. Would you have some idea why this happened? It is probably somehow related to the pagination state.

    DataStax JIRA | 11 months ago | Jakub Janecek
    com.datastax.driver.core.exceptions.ProtocolError: An unexpected error occurred on host /10.10.11.121:9042. This is a bug in library, please report: Invalid value for the paging state
  2. 0

    We have encountered the following exception during scan of our Cassandra cluster: {code} 13:19:03.903 ERROR c.a.u.d.LoggingLatencyTracker$ [cassandra-nio-worker-4]: Exception when making query to /10.10.11.121:9042, latency was 7 ms (LoggingLatencyTracker.scala:9) com.datastax.driver.core.exceptions.ProtocolError: An unexpected error occurred on host /10.10.11.121:9042. This is a bug in library, please report: Invalid value for the paging state at com.datastax.driver.core.Responses$Error.asException(Responses.java:92) [cassandra-driver-core-2.1.9.jar:na] at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:433) [cassandra-driver-core-2.1.9.jar:na] at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1009) [cassandra-driver-core-2.1.9.jar:na] at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:932) [cassandra-driver-core-2.1.9.jar:na] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) [netty-handler-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] {code} Because the scan is taking several weeks we need it to be restartable. Therefore we are using {{resultSet.getExecutionInfo.getPagingStateUnsafe}} to retrieve the current paging state and store it in a file. If the process is restarted it will use this paging state to continue from where it stopped before via {{Statement.setPagingStateUnsafe(state)}}. We are running the scan on several nodes at once to speed it up so the query looks like this: {code} select key, column, value, token(key) from keyspace.cf where token(key) >= 0 and token(key) < 100 {code} The process was running fine for several days until it failed on one of the nodes with this exception. The reason is unknown and we would like you to help us debug it (please, let me know how we can help you) and in the end possibly fix a bug in the driver. There is another issue we encountered. All of our scans (as I mentioned we run several parallel scans at once on different token ranges to speed things up) finished "successfully" at certain point. We our computing the progress based on the current token (we know the start and the end so we can compute how far we are) and all the scans finished at around 30%. They refused to continue even after restart and the only option was to adjust the token range to continue. We know for sure there are more data to be scanned because using CQLSH we can list them. Would you have some idea why this happened? It is probably somehow related to the pagination state.

    DataStax JIRA | 11 months ago | Jakub Janecek
    com.datastax.driver.core.exceptions.ProtocolError: An unexpected error occurred on host /10.10.11.121:9042. This is a bug in library, please report: Invalid value for the paging state
  3. 0

    running 3.0.0 stress app against cassandra 2.0.1 will result in the following error: {noformat} Unexpected error: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level at com.datastax.driver.core.exceptions.ProtocolError.copy(ProtocolError.java:65) at com.datastax.driver.core.exceptions.ProtocolError.copy(ProtocolError.java:27) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39) at com.datastax.driver.stress.Generators$2.prepare(Generators.java:88) at com.datastax.driver.stress.Stress$Stresser.prepare(Stress.java:156) at com.datastax.driver.stress.Stress.main(Stress.java:244) Caused by: com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level at com.datastax.driver.core.Responses$Error.asException(Responses.java:110) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179) at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:184) at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:798) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:617) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) at java.lang.Thread.run(Thread.java:745) {noformat} i can resolve this error in my sample application by adding QueryOptions().setConsistencyLevel(ConsistencyLevel.ONE)): {code} Cluster cluster = new Cluster.Builder() .addContactPoints(String.valueOf(options.valueOf("ip"))) .withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.ONE)) .withPoolingOptions(pools) .withSocketOptions(new SocketOptions().setTcpNoDelay(true)) .build(); {code}

    DataStax JIRA | 9 months ago | Alram Lechner
    com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    running 3.0.0 stress app against cassandra 2.0.1 will result in the following error: {noformat} Unexpected error: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level at com.datastax.driver.core.exceptions.ProtocolError.copy(ProtocolError.java:65) at com.datastax.driver.core.exceptions.ProtocolError.copy(ProtocolError.java:27) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39) at com.datastax.driver.stress.Generators$2.prepare(Generators.java:88) at com.datastax.driver.stress.Stress$Stresser.prepare(Stress.java:156) at com.datastax.driver.stress.Stress.main(Stress.java:244) Caused by: com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level at com.datastax.driver.core.Responses$Error.asException(Responses.java:110) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179) at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:184) at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:798) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:617) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) at java.lang.Thread.run(Thread.java:745) {noformat} i can resolve this error in my sample application by adding QueryOptions().setConsistencyLevel(ConsistencyLevel.ONE)): {code} Cluster cluster = new Cluster.Builder() .addContactPoints(String.valueOf(options.valueOf("ip"))) .withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.ONE)) .withPoolingOptions(pools) .withSocketOptions(new SocketOptions().setTcpNoDelay(true)) .build(); {code}

    DataStax JIRA | 9 months ago | Alram Lechner
    com.datastax.driver.core.exceptions.ProtocolError: An unexpected protocol error occurred on host localhost/127.0.0.1:9042. This is a bug in this library, please report: Unknown code 10 for a consistency level

    Root Cause Analysis

    1. com.datastax.driver.core.exceptions.ProtocolError

      An unexpected error occurred on host /10.10.11.121:9042. This is a bug in library, please report: Invalid value for the paging state

      at com.datastax.driver.core.Responses$Error.asException()
    2. DataStax Java Driver for Apache Cassandra - Core
      Connection$Dispatcher.channelRead0
      1. com.datastax.driver.core.Responses$Error.asException(Responses.java:92)[cassandra-driver-core-2.1.9.jar:na]
      2. com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:433)[cassandra-driver-core-2.1.9.jar:na]
      3. com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1009)[cassandra-driver-core-2.1.9.jar:na]
      4. com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:932)[cassandra-driver-core-2.1.9.jar:na]
      4 frames
    3. Netty
      SingleThreadEventExecutor$2.run
      1. io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      2. io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      3. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      4. io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)[netty-handler-4.0.27.Final.jar:4.0.27.Final]
      5. io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      6. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      7. io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)[netty-codec-4.0.27.Final.jar:4.0.27.Final]
      8. io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      9. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      10. io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)[netty-codec-4.0.27.Final.jar:4.0.27.Final]
      11. io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      12. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      13. io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      14. io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      15. io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      16. io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      17. io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      18. io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[netty-transport-4.0.27.Final.jar:4.0.27.Final]
      19. io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[netty-common-4.0.27.Final.jar:4.0.27.Final]
      19 frames
    4. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)[na:1.8.0_40]
      1 frame