com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)

DataStax JIRA | Joaquin Casares | 3 years ago
  1. 0

    We're still seeing the same issue with isUp() returning too early. Here are the full stacktraces behind run #116: {CODE} Tests run: 96, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 2,659.05 sec <<< FAILURE! dcAwareRoundRobinTestWithOneRemoteHost(com.datastax.driver.core.LoadBalancingPolicyTest) Time elapsed: 103.163 sec <<< FAILURE! java.lang.IllegalStateException: 127.0.1.5 is not UP after 30s at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:333) at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:259) at com.datastax.driver.core.LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost(LoadBalancingPolicyTest.java:160) reprepareOnNewlyUpNodeNoKeyspaceTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 18.34 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:156) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest(PreparedStatementTest.java:322) reprepareOnNewlyUpNodeTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 0 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:151) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.Session.execute(Session.java:80) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:288) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:314) exponentialReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 71.494 sec <<< FAILURE! java.lang.AssertionError: Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:202) at com.datastax.driver.core.ReconnectionPolicyTest.exponentialReconnectionPolicyTest(ReconnectionPolicyTest.java:82) listenerTest(com.datastax.driver.core.StateListenerTest) Time elapsed: 80.464 sec <<< FAILURE! java.lang.AssertionError: expected [1] but found [2] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertEquals(Assert.java:123) at org.testng.Assert.assertEquals(Assert.java:370) at org.testng.Assert.assertEquals(Assert.java:380) at com.datastax.driver.core.StateListenerTest.listenerTest(StateListenerTest.java:47) Results : Failed tests: LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost:160 » IllegalState PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest:322->reprepareOnNewlyUpNodeTest:303 » NoHostAvailable PreparedStatementTest.reprepareOnNewlyUpNodeTest:314->reprepareOnNewlyUpNodeTest:288 » NoHostAvailable ReconnectionPolicyTest.exponentialReconnectionPolicyTest:82->reconnectionPolicyTest:202 Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] StateListenerTest.listenerTest:47 expected [1] but found [2] {CODE} The NoHostAvailableException is the exception that you can use to debug the issue at isUp(). Here is one example: {CODE} at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) waitFor(CCMBridge.IP_PREFIX + "1", cluster); try { assertEquals(session.execute(ps.bind("124")).one().getInt("i"), 18); } {CODE} Which needs this portion in order to return from waitFor(): {CODE} in TestUtils.java private static boolean testHost(Host host, boolean testForDown) { return testForDown ? !host.isUp() : host.isUp(); } {CODE} There you'll see that isUp() should have waited until the binary protocol is up, before continuing on. Let me know if you need any additional information.

    DataStax JIRA | 3 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  2. 0

    We're still seeing the same issue with isUp() returning too early. Here are the full stacktraces behind run #116: {CODE} Tests run: 96, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 2,659.05 sec <<< FAILURE! dcAwareRoundRobinTestWithOneRemoteHost(com.datastax.driver.core.LoadBalancingPolicyTest) Time elapsed: 103.163 sec <<< FAILURE! java.lang.IllegalStateException: 127.0.1.5 is not UP after 30s at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:333) at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:259) at com.datastax.driver.core.LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost(LoadBalancingPolicyTest.java:160) reprepareOnNewlyUpNodeNoKeyspaceTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 18.34 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:156) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest(PreparedStatementTest.java:322) reprepareOnNewlyUpNodeTest(com.datastax.driver.core.PreparedStatementTest) Time elapsed: 0 sec <<< FAILURE! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.Session$Manager.execute(Session.java:474) at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510) at com.datastax.driver.core.Session.executeAsync(Session.java:151) at com.datastax.driver.core.Session.execute(Session.java:111) at com.datastax.driver.core.Session.execute(Session.java:80) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:288) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:314) exponentialReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 71.494 sec <<< FAILURE! java.lang.AssertionError: Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:202) at com.datastax.driver.core.ReconnectionPolicyTest.exponentialReconnectionPolicyTest(ReconnectionPolicyTest.java:82) listenerTest(com.datastax.driver.core.StateListenerTest) Time elapsed: 80.464 sec <<< FAILURE! java.lang.AssertionError: expected [1] but found [2] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertEquals(Assert.java:123) at org.testng.Assert.assertEquals(Assert.java:370) at org.testng.Assert.assertEquals(Assert.java:380) at com.datastax.driver.core.StateListenerTest.listenerTest(StateListenerTest.java:47) Results : Failed tests: LoadBalancingPolicyTest.dcAwareRoundRobinTestWithOneRemoteHost:160 » IllegalState PreparedStatementTest.reprepareOnNewlyUpNodeNoKeyspaceTest:322->reprepareOnNewlyUpNodeTest:303 » NoHostAvailable PreparedStatementTest.reprepareOnNewlyUpNodeTest:314->reprepareOnNewlyUpNodeTest:288 » NoHostAvailable ReconnectionPolicyTest.exponentialReconnectionPolicyTest:82->reconnectionPolicyTest:202 Waited 32 seconds instead an expected 30 seconds wait expected [true] but found [false] StateListenerTest.listenerTest:47 expected [1] but found [2] {CODE} The NoHostAvailableException is the exception that you can use to debug the issue at isUp(). Here is one example: {CODE} at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:303) waitFor(CCMBridge.IP_PREFIX + "1", cluster); try { assertEquals(session.execute(ps.bind("124")).one().getInt("i"), 18); } {CODE} Which needs this portion in order to return from waitFor(): {CODE} in TestUtils.java private static boolean testHost(Host host, boolean testForDown) { return testForDown ? !host.isUp() : host.isUp(); } {CODE} There you'll see that isUp() should have waited until the binary protocol is up, before continuing on. Let me know if you need any additional information.

    DataStax JIRA | 3 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  3. 0

    http://cassci.datastax.com/view/y_Drivers/job/y_java-driver_2.0/117/testReport/com.datastax.driver.core/PreparedStatementTest/reprepareOnNewlyUpNodeTest/ I swapped out the offending line with: {CODE} try { Thread.sleep(180 * 1000); } catch (Exception e) {} {CODE} to get rid of any possibility of race conditions. Running `cassandraCluster.ring();` before and after the shutting down and starting up of the node shows the cluster switched to vnodes on restart. Not sure if this affects the test result. That's when the session.execute() command returns the following error: {CODE} com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.SessionManager.execute(SessionManager.java:368) at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:404) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:85) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:296) at com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:339) {CODE} Real world use case: If a cluster is running and a network partition happens, or the cluster really shuts down, on restart, it doesn't appear as though the driver will connect again without manual intervention.

    DataStax JIRA | 2 years ago | Joaquin Casares
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassandra Write Timeouts

    GitHub | 4 weeks ago | nsoft
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
  6. 0

    Re: Execption writing on two cassandra tables NoHostAvailableException: All host(s) tried for query failed (no host was tried)

    spark-user | 2 years ago | Antonio Giambanco
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 61.0 failed 1 times, most recent failure: Lost task 0.0 in stage 61.0 (TID 81, localhost): java.io.IOException: Failed to prepare statement INSERT INTO "cassandrasink"."transaction" ("event_id", "isin", "security_type", "security_name", "date", "time", "price", "currency", "user_id", "quantity", "amount", "session_id") VALUES (:"event_id", :"isin", :"security_type", :"security_name", :"date", :"time", :"price", :"currency", :"user_id", :"quantity", :"amount", :"session_id"): All host(s) tried for query failed (no host was tried)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.datastax.driver.core.exceptions.NoHostAvailableException

      All host(s) tried for query failed (no host was tried)

      at com.datastax.driver.core.RequestHandler.sendRequest()
    2. DataStax Java Driver for Apache Cassandra - Core
      PreparedStatementTest.reprepareOnNewlyUpNodeTest
      1. com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103)
      2. com.datastax.driver.core.Session$Manager.execute(Session.java:474)
      3. com.datastax.driver.core.Session$Manager.executeQuery(Session.java:510)
      4. com.datastax.driver.core.Session.executeAsync(Session.java:151)
      5. com.datastax.driver.core.Session.execute(Session.java:111)
      6. com.datastax.driver.core.Session.execute(Session.java:80)
      7. com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:288)
      8. com.datastax.driver.core.PreparedStatementTest.reprepareOnNewlyUpNodeTest(PreparedStatementTest.java:314)
      8 frames