com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)

DataStax JIRA | Joaquin Casares | 3 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    http://cassci.datastax.com/view/y_Drivers/job/y_java-driver_2.0/117/testReport/com.datastax.driver.core/PreparedStatementTest/reprepareOnNewlyUpNodeTest/ I swapped out the offending line with: {CODE} try { Thread.sleep(180 * 1000); } catch (Exception e) {} {CODE} to get rid of any possibility of race conditions. Running `cassandraCluster.ring();` before and after the shutting down and starting up of the node shows the cluster switched to vnodes on restart. Not sure if this affects the test result. That's when the session.execute() command returns the following error: {CODE} com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.SessionManager.execute(SessionManager.java:368) at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:404) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:85) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:296) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:339) {CODE} Real world use case: If a cluster is running and a network partition happens, or the cluster really shuts down, on restart, it doesn't appear as though the driver will connect again without manual intervention.

    DataStax JIRA | 3 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  2. 0

    We're still seeing the same issue with isUp() returning too early. Here are the full stacktraces behind run #116: {CODE} Tests run: 96, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 2,659.05 sec <<< FAILURE! dcAwareRoundRobinTestWithOneRemoteHost(com.datastax.driver.core.LoadBalancingPolicyTest) Time elapsed: 103.163 sec <<< FAILURE! java.lang.IllegalStateException: 127.0.1.5 is not UP after 30s at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:333) at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:259) at com.datastax.driver.core.LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost(LoadBalancingPolicyTest.java:160) reprepareOnNewlyUpNodeNoKeyspaceTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 18.34 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:156) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest(PreparedStatementTest.java:322) reprepareOnNewlyUpNodeTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 0 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:151) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.Session.execute(Session.java:80) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:288) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:314) exponentialReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 71.494 sec <<< FAILURE! java.lang.AssertionError: Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:202) at com.datastax.driver.core.ReconnectionPolicyTest.exponentialReconnectionPolicyTest(ReconnectionPolicyTest.java:82) listenerTest(com.datastax.driver.core.StateListenerTest) Time elapsed: 80.464 sec <<< FAILURE! java.lang.AssertionError: expected [1] but found [2] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertEquals(Assert.java:123) at org.testng.Assert.assertEquals(Assert.java:370) at org.testng.Assert.assertEquals(Assert.java:380) at com.datastax.driver.core.StateListenerTest.listenerTest(StateListenerTest.java:47) Results : Failed tests: LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost:160 » IllegalState PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest:322->reprepareOnNewlyUpNodeTest:303 » NoHostAvailable PreparedStatementTest.reprepareOnNewlyUpNodeTest:314->reprepareOnNewlyUpNodeTest:288 » NoHostAvailable ReconnectionPolicyTest.exponentialReconnectionPolicyTest:82->reconnectionPolicyTest:202 Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] StateListenerTest.listenerTest:47 expected [1] but found [2] {CODE} The NoHostAvailableException is the exception that you can use to debug the issue at isUp(). Here is one example: {CODE} at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) waitFor(CCMBridge.IP_PREFIX + "1", cluster); try { assertEquals(session.execute(ps.bind("124")).one().getInt("i"), 18); } {CODE} Which needs this portion in order to return from waitFor(): {CODE} in TestUtils.java private static boolean testHost(Host host, boolean testForDown) { return testForDown ? !host.isUp() : host.isUp(); } {CODE} There you'll see that isUp() should have waited until the binary protocol is up, before continuing on. Let me know if you need any additional information.

    DataStax JIRA | 4 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  3. 0

    We're still seeing the same issue with isUp() returning too early. Here are the full stacktraces behind run #116: {CODE} Tests run: 96, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 2,659.05 sec <<< FAILURE! dcAwareRoundRobinTestWithOneRemoteHost(com.datastax.driver.core.LoadBalancingPolicyTest) Time elapsed: 103.163 sec <<< FAILURE! java.lang.IllegalStateException: 127.0.1.5 is not UP after 30s at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:333) at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:259) at com.datastax.driver.core.LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost(LoadBalancingPolicyTest.java:160) reprepareOnNewlyUpNodeNoKeyspaceTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 18.34 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:156) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest(PreparedStatementTest.java:322) reprepareOnNewlyUpNodeTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 0 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:151) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.Session.execute(Session.java:80) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:288) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:314) exponentialReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 71.494 sec <<< FAILURE! java.lang.AssertionError: Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:202) at com.datastax.driver.core.ReconnectionPolicyTest.exponentialReconnectionPolicyTest(ReconnectionPolicyTest.java:82) listenerTest(com.datastax.driver.core.StateListenerTest) Time elapsed: 80.464 sec <<< FAILURE! java.lang.AssertionError: expected [1] but found [2] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertEquals(Assert.java:123) at org.testng.Assert.assertEquals(Assert.java:370) at org.testng.Assert.assertEquals(Assert.java:380) at com.datastax.driver.core.StateListenerTest.listenerTest(StateListenerTest.java:47) Results : Failed tests: LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost:160 » IllegalState PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest:322->reprepareOnNewlyUpNodeTest:303 » NoHostAvailable PreparedStatementTest.reprepareOnNewlyUpNodeTest:314->reprepareOnNewlyUpNodeTest:288 » NoHostAvailable ReconnectionPolicyTest.exponentialReconnectionPolicyTest:82->reconnectionPolicyTest:202 Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] StateListenerTest.listenerTest:47 expected [1] but found [2] {CODE} The NoHostAvailableException is the exception that you can use to debug the issue at isUp(). Here is one example: {CODE} at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) waitFor(CCMBridge.IP_PREFIX + "1", cluster); try { assertEquals(session.execute(ps.bind("124")).one().getInt("i"), 18); } {CODE} Which needs this portion in order to return from waitFor(): {CODE} in TestUtils.java private static boolean testHost(Host host, boolean testForDown) { return testForDown ? !host.isUp() : host.isUp(); } {CODE} There you'll see that isUp() should have waited until the binary protocol is up, before continuing on. Let me know if you need any additional information.

    DataStax JIRA | 4 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.datastax.driver.core.exceptions.NoHostAvailableException

      All host(s) tried for query failed (no host was tried)

      at com.datastax.driver.core.RequestHandler.sendRequest()
    2. DataStax Java Driver for Apache Cassandra - Core
      PreparedStatementTest.reprepareOnNewlyUpNodeTest
      1. com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103)
      2. com.datastax.driver.core.SessionManager.execute(SessionManager.java:368)
      3. com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:404)
      4. com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:85)
      5. com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
      6. com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
      7. com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:296)
      8. com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:339)
      8 frames