com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.

Apache's JIRA Issue Tracker | Vishy Kasar | 2 years ago
  1. 0

    We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java (line 224) Error occurred during processing of message. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:116) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more

    Apache's JIRA Issue Tracker | 2 years ago | Vishy Kasar
    com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  2. 0

    Fail to reconnect to other nodes after intermittent network failure

    Google Groups | 2 years ago | Jiri Horky
    org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  3. 0

    Fail to reconnect to other nodes after intermittent network failure

    Google Groups | 2 years ago | Jiri Horky
    org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassandra Cluster in AWS Multi Region VPC

    Stack Overflow | 2 years ago | THClem
    com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  6. 0

    C* 1.2.15 Decommission issues

    Google Groups | 3 years ago | Russell Bradberry
    java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.cassandra.exceptions.ReadTimeoutException

      Operation timed out - received only 0 responses.

      at org.apache.cassandra.service.ReadCallback.get()
    2. org.apache.cassandra
      ClientState$1.load
      1. org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105)
      2. org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943)
      3. org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828)
      4. org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140)
      5. org.apache.cassandra.auth.Auth.selectUser(Auth.java:245)
      6. org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84)
      7. org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
      8. org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68)
      9. org.apache.cassandra.service.ClientState$1.load(ClientState.java:278)
      10. org.apache.cassandra.service.ClientState$1.load(ClientState.java:275)
      10 frames
    3. Guava
      LocalCache$LocalLoadingCache.get
      1. com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
      2. com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
      3. com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
      4. com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
      5. com.google.common.cache.LocalCache.get(LocalCache.java:3990)
      6. com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
      7. com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
      7 frames
    4. org.apache.cassandra
      QueryProcessor.process
      1. org.apache.cassandra.service.ClientState.authorize(ClientState.java:292)
      2. org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172)
      3. org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165)
      4. org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149)
      5. org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:116)
      6. org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102)
      7. org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113)
      7 frames
    5. Apache Cassandra
      Cassandra$Processor$execute_cql3_query.getResult
      1. org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735)
      2. org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162)
      3. org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150)
      3 frames
    6. Apache Thrift
      TBaseProcessor.process
      1. org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
      2. org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
      2 frames
    7. Apache Cassandra
      CustomTThreadPoolServer$WorkerProcess.run
      1. org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206)
      1 frame
    8. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      3. java.lang.Thread.run(Thread.java:722)
      3 frames