com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (5 replica were required but only 4 acknowledged the write)

DataStax JIRA | Joaquin Casares | 3 years ago
  1. 0

    I was running the duration_test (with a failure bot) on a 3 node cluster with an RF of 3. Before starting the test, I did run a cassandra-shuffle, then cleared and disabled it. The tokens did change later: {CODE} Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.191.123.72 464.68 KB 256 99.2% e9da1abf-0aa6-4696-a918-3e256f3a76a8 1c DN 10.190.61.162 73.02 KB 1 0.6% becdaabc-0e41-4d9d-abd9-d14a77c52881 1c UN 10.191.13.148 64.07 KB 1 0.1% 14576211-08b6-4b92-84d9-1a50dc6620df 1c {CODE} Then the required replicas increased to > 3. They did rotate between 4 and 5 replicas: {CODE} Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (4 replica were required but only 3 acknowledged the write) at com.datastax.driver.core.ResultSetFuture.convertException(ResultSetFuture.java:263) at com.datastax.driver.core.ResultSetFuture$ResponseCallback.onSet(ResultSetFuture.java:109) at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:200) at com.datastax.driver.core.RequestHandler.onSet(RequestHandler.java:328) at com.datastax.driver.core.Connection$Dispatcher.messageReceived(Connection.java:472) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:68) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ... 3 more com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (5 replica were required but only 4 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:51) at com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:234) at com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:165) at com.datastax.driver.core.Session.execute(Session.java:106) at com.datastax.cassandra.examples.durationtest.TestUtils.executeUntilPass(TestUtils.java:56) at com.datastax.cassandra.examples.durationtest.Write.queryBuilder(Write.java:64) at com.datastax.cassandra.examples.durationtest.Write.write(Write.java:39) at com.datastax.cassandra.examples.durationtest.DurationTest$SuiteRunnable.run(DurationTest.java:107) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) {CODE}

    DataStax JIRA | 3 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (5 replica were required but only 4 acknowledged the write)
  2. 0

    I was running the duration_test (with a failure bot) on a 3 node cluster with an RF of 3. Before starting the test, I did run a cassandra-shuffle, then cleared and disabled it. The tokens did change later: {CODE} Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.191.123.72 464.68 KB 256 99.2% e9da1abf-0aa6-4696-a918-3e256f3a76a8 1c DN 10.190.61.162 73.02 KB 1 0.6% becdaabc-0e41-4d9d-abd9-d14a77c52881 1c UN 10.191.13.148 64.07 KB 1 0.1% 14576211-08b6-4b92-84d9-1a50dc6620df 1c {CODE} Then the required replicas increased to > 3. They did rotate between 4 and 5 replicas: {CODE} Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (4 replica were required but only 3 acknowledged the write) at com.datastax.driver.core.ResultSetFuture.convertException(ResultSetFuture.java:263) at com.datastax.driver.core.ResultSetFuture$ResponseCallback.onSet(ResultSetFuture.java:109) at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:200) at com.datastax.driver.core.RequestHandler.onSet(RequestHandler.java:328) at com.datastax.driver.core.Connection$Dispatcher.messageReceived(Connection.java:472) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:68) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ... 3 more com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (5 replica were required but only 4 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:51) at com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:234) at com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:165) at com.datastax.driver.core.Session.execute(Session.java:106) at com.datastax.cassandra.examples.durationtest.TestUtils.executeUntilPass(TestUtils.java:56) at com.datastax.cassandra.examples.durationtest.Write.queryBuilder(Write.java:64) at com.datastax.cassandra.examples.durationtest.Write.write(Write.java:39) at com.datastax.cassandra.examples.durationtest.DurationTest$SuiteRunnable.run(DurationTest.java:107) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) {CODE}

    DataStax JIRA | 3 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (5 replica were required but only 4 acknowledged the write)
  3. 0

    Cassandra Frequent Read Write Timeouts

    Stack Overflow | 3 years ago | user2572801
    com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassandra Write Time out error and no rollback occured

    Stack Overflow | 9 months ago | Rakesh Singh Rana
    com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.datastax.driver.core.exceptions.WriteTimeoutException

      Cassandra timeout during write query at consistency QUORUM (5 replica were required but only 4 acknowledged the write)

      at com.datastax.driver.core.exceptions.WriteTimeoutException.copy()
    2. DataStax Java Driver for Apache Cassandra - Core
      Session.execute
      1. com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:51)
      2. com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:234)
      3. com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:165)
      4. com.datastax.driver.core.Session.execute(Session.java:106)
      4 frames
    3. com.datastax.cassandra
      DurationTest$SuiteRunnable.run
      1. com.datastax.cassandra.examples.durationtest.TestUtils.executeUntilPass(TestUtils.java:56)
      2. com.datastax.cassandra.examples.durationtest.Write.queryBuilder(Write.java:64)
      3. com.datastax.cassandra.examples.durationtest.Write.write(Write.java:39)
      4. com.datastax.cassandra.examples.durationtest.DurationTest$SuiteRunnable.run(DurationTest.java:107)
      4 frames
    4. Java RT
      Thread.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      2. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      3. java.util.concurrent.FutureTask.run(FutureTask.java:138)
      4. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      5. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      6. java.lang.Thread.run(Thread.java:662)
      6 frames