com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Problem while setting keyspace, got ERROR INVALID: Keyspace 'xxx' does not exist as response

DataStax JIRA | Vishy Kasar | 2 years ago
  1. 0

    We have few clients connecting to nodes of another cluster due to some operational mistakes. We can detect such issues sooner if we log the invalid keyspave below at -ERROR- WARN level. (_*Note: level changed after the issue was open, see discussion below*_) {code} 2014/10/20 10:43:50.968  bld14E37 svn16344 tags[remaining] TRACE [New I/O worker #51] com.datastax.driver.core.Connection - [host1/ip1:9042-41102] received: ERROR INVALID: Keyspace 'xxx' does not exist 2014/10/20 10:43:50.968  bld14E37 svn16344 tags[remaining] TRACE [New I/O worker #42] com.datastax.driver.core.Connection - [host/ip:9042-2] request sent successfully 2014/10/20 10:43:50.968  bld14E37 svn16344 tags[remaining] DEBUG [Cassandra Java Driver worker-14] com.datastax.driver.core.Connection - Defuncting connection to host1/ip1:9042 com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Problem while setting keyspace, got ERROR INVALID: Keyspace 'xxx' does not exist as response         at com.datastax.driver.core.Connection.setKeyspace(Connection.java:287)         at com.datastax.driver.core.Cluster$Manager.prepareAllQueries(Cluster.java:1723)         at com.datastax.driver.core.Cluster$Manager.onUp(Cluster.java:1295)         at com.datastax.driver.core.Cluster$Manager.access$900(Cluster.java:1064)         at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1406)         at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744) {code}

    DataStax JIRA | 2 years ago | Vishy Kasar
    com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Problem while setting keyspace, got ERROR INVALID: Keyspace 'xxx' does not exist as response
  2. 0

    We have few clients connecting to nodes of another cluster due to some operational mistakes. We can detect such issues sooner if we log the invalid keyspave below at -ERROR- WARN level. (_*Note: level changed after the issue was open, see discussion below*_) {code} 2014/10/20 10:43:50.968  bld14E37 svn16344 tags[remaining] TRACE [New I/O worker #51] com.datastax.driver.core.Connection - [host1/ip1:9042-41102] received: ERROR INVALID: Keyspace 'xxx' does not exist 2014/10/20 10:43:50.968  bld14E37 svn16344 tags[remaining] TRACE [New I/O worker #42] com.datastax.driver.core.Connection - [host/ip:9042-2] request sent successfully 2014/10/20 10:43:50.968  bld14E37 svn16344 tags[remaining] DEBUG [Cassandra Java Driver worker-14] com.datastax.driver.core.Connection - Defuncting connection to host1/ip1:9042 com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Problem while setting keyspace, got ERROR INVALID: Keyspace 'xxx' does not exist as response         at com.datastax.driver.core.Connection.setKeyspace(Connection.java:287)         at com.datastax.driver.core.Cluster$Manager.prepareAllQueries(Cluster.java:1723)         at com.datastax.driver.core.Cluster$Manager.onUp(Cluster.java:1295)         at com.datastax.driver.core.Cluster$Manager.access$900(Cluster.java:1064)         at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1406)         at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744) {code}

    DataStax JIRA | 2 years ago | Vishy Kasar
    com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Problem while setting keyspace, got ERROR INVALID: Keyspace 'xxx' does not exist as response
  3. 0

    Timeout during the first connection: {code} 2015-11-22 21:58:32,870 ERROR [stderr] (Thread-7) com.datastax.driver.core.ConnectionException: [/x.x.x.x:9042] Timeout while setting keyspace on connection 2015-11-22 21:58:32,870 ERROR [stderr] (Thread-7) at com.datastax.driver.core.Connection.setKeyspace(Connection.java:451) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.HostConnectionPool.borrowConnection(HostConnectionPool.java:277) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:296) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:276) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.run(RequestHandler.java:377) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-22 21:58:32,872 ERROR [stderr] (Thread-7) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2015-11-22 21:58:32,872 ERROR [stderr] (Thread-7) at java.lang.Thread.run(Thread.java:745) {code} After after the reconnect, works. :/

    DataStax JIRA | 1 year ago | Gábor AUTH
    com.datastax.driver.core.ConnectionException: [/x.x.x.x:9042] Timeout while setting keyspace on connection 2015-11-22 21:58:32,870 ERROR [stderr] (Thread-7) at com.datastax.driver.core.Connection.setKeyspace(Connection.java:451)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Timeout during the first connection: {code} 2015-11-22 21:58:32,870 ERROR [stderr] (Thread-7) com.datastax.driver.core.ConnectionException: [/x.x.x.x:9042] Timeout while setting keyspace on connection 2015-11-22 21:58:32,870 ERROR [stderr] (Thread-7) at com.datastax.driver.core.Connection.setKeyspace(Connection.java:451) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.HostConnectionPool.borrowConnection(HostConnectionPool.java:277) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:296) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:276) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.run(RequestHandler.java:377) 2015-11-22 21:58:32,871 ERROR [stderr] (Thread-7) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-22 21:58:32,872 ERROR [stderr] (Thread-7) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2015-11-22 21:58:32,872 ERROR [stderr] (Thread-7) at java.lang.Thread.run(Thread.java:745) {code} After after the reconnect, works. :/

    DataStax JIRA | 1 year ago | Gábor AUTH
    com.datastax.driver.core.ConnectionException: [/x.x.x.x:9042] Timeout while setting keyspace on connection 2015-11-22 21:58:32,870 ERROR [stderr] (Thread-7) at com.datastax.driver.core.Connection.setKeyspace(Connection.java:451)
  6. 0

    There may have been temporary issues in host2 system.peers table missing entry for hots1. When this happened, it appears java driver gave up on host1. Right after this message, when I looked up in the host2 system.peers, the entry was present. OnUp() was never called on host1 again. On long running systems, we should strive to call onUp as soon as possible. Else we will end up with dwindling host situation on the client side. 2014-10-02 09:15:43 ERROR Session:279 - Error creating pool to host1/ip1:9042 com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out) at com.datastax.driver.core.Connection.initializeTransport(Connection.java:161) at com.datastax.driver.core.Connection.<init>(Connection.java:113) at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:32) at com.datastax.driver.core.Connection$Factory.open(Connection.java:516) at com.datastax.driver.core.HostConnectionPool.<init>(HostConnectionPool.java:85) at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:241) at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:42) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:273) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:265) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:774) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:828) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ... 1 more Oct 2, 2014 9:15:43 AM org.jboss.netty.util.HashedWheelTimer WARNING: An exception was thrown by TimerTask. java.lang.NullPointerException at com.datastax.driver.core.PooledConnection.release(PooledConnection.java:41) at com.datastax.driver.core.Connection$ResponseHandler.cancelHandler(Connection.java:821) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:829) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) at java.lang.Thread.run(Thread.java:695) 2014-10-02 09:15:57 ERROR ControlConnection:420 - No row found for host host1/ip1 in host2/ip2:9042's peers system table. host1/ip1 will be ignored.

    DataStax JIRA | 2 years ago | Vishy Kasar
    com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.datastax.driver.core.ConnectionException

      [host1/ip1:9042] Problem while setting keyspace, got ERROR INVALID: Keyspace 'xxx' does not exist as response

      at com.datastax.driver.core.Connection.setKeyspace()
    2. DataStax Java Driver for Apache Cassandra - Core
      ExceptionCatchingRunnable.run
      1. com.datastax.driver.core.Connection.setKeyspace(Connection.java:287)
      2. com.datastax.driver.core.Cluster$Manager.prepareAllQueries(Cluster.java:1723)
      3. com.datastax.driver.core.Cluster$Manager.onUp(Cluster.java:1295)
      4. com.datastax.driver.core.Cluster$Manager.access$900(Cluster.java:1064)
      5. com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1406)
      6. com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)
      6 frames
    3. Java RT
      Thread.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      2. java.util.concurrent.FutureTask.run(FutureTask.java:262)
      3. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      4. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      5. java.lang.Thread.run(Thread.java:744)
      5 frames