com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out

DataStax JIRA | Mohammed Guller | 2 years ago
  1. 0

    While running an application that reads data from Cassandra 2.1.0 using Java driver 2.1.2 or 2.1.3, the driver starts throwing the error messages shown below. There is no connectivity issue since the driver is able to connect and read data when the application is launched. These error messages are logged after the application has been running for some time. {noformat} 21:15:59 WARN c.d.d.c.Connection Forcing termination of Connection[hostname/ip:9042-1, inFlight=5, closed=true]. This should not happen and is likely a bug, please report. 21:17:29 ERROR c.d.d.c.Session Error creating pool to hostname/ip:9042 com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:140) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:855) ~[cassandra-driver-core-2.1.2.jar:na] at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) ~[netty-3.9.0.Final.jar:na] at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) ~[netty-3.9.0.Final.jar:na] at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[netty-3.9.0.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.9.0.Final.jar:na] Wrapped by: com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out) at com.datastax.driver.core.Connection.initializeTransport(Connection.java:166) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection.<init>(Connection.java:113) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:32) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection$Factory.open(Connection.java:521) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.SingleConnectionPool.<init>(SingleConnectionPool.java:76) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.HostConnectionPool.newInstance(HostConnectionPool.java:35) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.SessionManager$2.call(SessionManager.java:199) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.SessionManager$2.call(SessionManager.java:192) [cassandra-driver-core-2.1.2.jar:na] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_03] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_03] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:na] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) [na:1.7.0_03] at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) [guava-16.0.1.jar:na] at com.datastax.driver.core.SessionManager.forceRenewPool(SessionManager.java:192) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Cluster$Manager.onUp(Cluster.java:1386) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Cluster$Manager.access$1000(Cluster.java:1110) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1474) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [cassandra-driver-core-2.1.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_03] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_03] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_03] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_03] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_03] at java.lang.Thread.run(Thread.java:722) [na:1.7.0_03] 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:39:30 ERROR c.d.d.c.Session Error creating pool to hostname/ip:9042 com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:797) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:855) ~[cassandra-driver-core-2.1.2.jar:na] {noformat}

    DataStax JIRA | 2 years ago | Mohammed Guller
    com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out
  2. 0

    While running an application that reads data from Cassandra 2.1.0 using Java driver 2.1.2 or 2.1.3, the driver starts throwing the error messages shown below. There is no connectivity issue since the driver is able to connect and read data when the application is launched. These error messages are logged after the application has been running for some time. {noformat} 21:15:59 WARN c.d.d.c.Connection Forcing termination of Connection[hostname/ip:9042-1, inFlight=5, closed=true]. This should not happen and is likely a bug, please report. 21:17:29 ERROR c.d.d.c.Session Error creating pool to hostname/ip:9042 com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:140) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:855) ~[cassandra-driver-core-2.1.2.jar:na] at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) ~[netty-3.9.0.Final.jar:na] at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) ~[netty-3.9.0.Final.jar:na] at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[netty-3.9.0.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.9.0.Final.jar:na] Wrapped by: com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out) at com.datastax.driver.core.Connection.initializeTransport(Connection.java:166) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection.<init>(Connection.java:113) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:32) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection$Factory.open(Connection.java:521) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.SingleConnectionPool.<init>(SingleConnectionPool.java:76) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.HostConnectionPool.newInstance(HostConnectionPool.java:35) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.SessionManager$2.call(SessionManager.java:199) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.SessionManager$2.call(SessionManager.java:192) [cassandra-driver-core-2.1.2.jar:na] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_03] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_03] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:na] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) [na:1.7.0_03] at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) [guava-16.0.1.jar:na] at com.datastax.driver.core.SessionManager.forceRenewPool(SessionManager.java:192) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Cluster$Manager.onUp(Cluster.java:1386) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Cluster$Manager.access$1000(Cluster.java:1110) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1474) [cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [cassandra-driver-core-2.1.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_03] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_03] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_03] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_03] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_03] at java.lang.Thread.run(Thread.java:722) [na:1.7.0_03] 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:30:22 WARN c.d.d.c.Connection Timeout while setting keyspace on connection to hostname/ip:9042. This should not happen but is not critical (it will retried) 21:39:30 ERROR c.d.d.c.Session Error creating pool to hostname/ip:9042 com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:797) ~[cassandra-driver-core-2.1.2.jar:na] at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:855) ~[cassandra-driver-core-2.1.2.jar:na] {noformat}

    DataStax JIRA | 2 years ago | Mohammed Guller
    com.datastax.driver.core.ConnectionException: [hostname/ip:9042] Operation timed out
  3. 0

    Spark cassandra connector connection error , no more host to try

    Stack Overflow | 2 years ago | Shashank Shukla
    com.datastax.driver.core.ConnectionException: [/192.168.1.26:9042] Operation timed out
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    There may have been temporary issues in host2 system.peers table missing entry for hots1. When this happened, it appears java driver gave up on host1. Right after this message, when I looked up in the host2 system.peers, the entry was present. OnUp() was never called on host1 again. On long running systems, we should strive to call onUp as soon as possible. Else we will end up with dwindling host situation on the client side. 2014-10-02 09:15:43 ERROR Session:279 - Error creating pool to host1/ip1:9042 com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out) at com.datastax.driver.core.Connection.initializeTransport(Connection.java:161) at com.datastax.driver.core.Connection.<init>(Connection.java:113) at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:32) at com.datastax.driver.core.Connection$Factory.open(Connection.java:516) at com.datastax.driver.core.HostConnectionPool.<init>(HostConnectionPool.java:85) at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:241) at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:42) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:273) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:265) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:774) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:828) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ... 1 more Oct 2, 2014 9:15:43 AM org.jboss.netty.util.HashedWheelTimer WARNING: An exception was thrown by TimerTask. java.lang.NullPointerException at com.datastax.driver.core.PooledConnection.release(PooledConnection.java:41) at com.datastax.driver.core.Connection$ResponseHandler.cancelHandler(Connection.java:821) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:829) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) at java.lang.Thread.run(Thread.java:695) 2014-10-02 09:15:57 ERROR ControlConnection:420 - No row found for host host1/ip1 in host2/ip2:9042's peers system table. host1/ip1 will be ignored.

    DataStax JIRA | 2 years ago | Vishy Kasar
    com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out)
  6. 0

    There may have been temporary issues in host2 system.peers table missing entry for hots1. When this happened, it appears java driver gave up on host1. Right after this message, when I looked up in the host2 system.peers, the entry was present. OnUp() was never called on host1 again. On long running systems, we should strive to call onUp as soon as possible. Else we will end up with dwindling host situation on the client side. 2014-10-02 09:15:43 ERROR Session:279 - Error creating pool to host1/ip1:9042 com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out) at com.datastax.driver.core.Connection.initializeTransport(Connection.java:161) at com.datastax.driver.core.Connection.<init>(Connection.java:113) at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:32) at com.datastax.driver.core.Connection$Factory.open(Connection.java:516) at com.datastax.driver.core.HostConnectionPool.<init>(HostConnectionPool.java:85) at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:241) at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:42) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:273) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:265) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Caused by: com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:774) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:828) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ... 1 more Oct 2, 2014 9:15:43 AM org.jboss.netty.util.HashedWheelTimer WARNING: An exception was thrown by TimerTask. java.lang.NullPointerException at com.datastax.driver.core.PooledConnection.release(PooledConnection.java:41) at com.datastax.driver.core.Connection$ResponseHandler.cancelHandler(Connection.java:821) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:829) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) at org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) at java.lang.Thread.run(Thread.java:695) 2014-10-02 09:15:57 ERROR ControlConnection:420 - No row found for host host1/ip1 in host2/ip2:9042's peers system table. host1/ip1 will be ignored.

    DataStax JIRA | 2 years ago | Vishy Kasar
    com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [host1/ip1:9042] Operation timed out)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.datastax.driver.core.ConnectionException

      [hostname/ip:9042] Operation timed out

      at com.datastax.driver.core.DefaultResultSetFuture.onTimeout()
    2. DataStax Java Driver for Apache Cassandra - Core
      Connection$ResponseHandler$1.run
      1. com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:140)[cassandra-driver-core-2.1.2.jar:na]
      2. com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:855)[cassandra-driver-core-2.1.2.jar:na]
      2 frames
    3. Netty
      ThreadRenamingRunnable.run
      1. org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)[netty-3.9.0.Final.jar:na]
      2. org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)[netty-3.9.0.Final.jar:na]
      3. org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)[netty-3.9.0.Final.jar:na]
      4. org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)[netty-3.9.0.Final.jar:na]
      4 frames