java.lang.IllegalStateException: 127.0.1.2 is not DOWN and is still part of the cluster after 20s

DataStax JIRA | Joaquin Casares | 4 years ago
  1. 0

    https://github.com/joaquincasares/java-driver/commit/a2a3f8890fc32ce4009c7889c90f34eccee48c14 This is unlike the issues that we've had before. Since the last issue was patched everything was working fine, aside from the race conditions that would last at most a few seconds. This new issue is that the waitFor* code can take as long as 250+ seconds before a node comes online. A 200 second delay handled a large number of cases, yet a 250 second delay handles all but typically 1 or 2. This also impacts the overall time it takes for tests to complete. The following issues tend to occur within the Reconnection and Retry Policy tests due to this new increase as well: {CODE} constantReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 78.054 sec <<< FAILURE! java.lang.AssertionError: Waited 70 seconds instead an expected 40 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:161) at com.datastax.driver.core.ReconnectionPolicyTest.constantReconnectionPolicyTest(ReconnectionPolicyTest.java:119) exponentialReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 70.081 sec <<< FAILURE! java.lang.AssertionError: Waited 62 seconds instead an expected 30 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:161) at com.datastax.driver.core.ReconnectionPolicyTest.exponentialReconnectionPolicyTest(ReconnectionPolicyTest.java:82) defaultLoggingPolicy(com.datastax.driver.core.RetryPolicyTest) Time elapsed: 430.156 sec <<< FAILURE! java.lang.IllegalStateException: 127.0.1.2 is not DOWN after 250s at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:337) at com.datastax.driver.core.TestUtils.waitForDownWithWait(TestUtils.java:275) at com.datastax.driver.core.RetryPolicyTest.defaultPolicyTest(RetryPolicyTest.java:219) at com.datastax.driver.core.RetryPolicyTest.defaultLoggingPolicy(RetryPolicyTest.java:76) {CODE}

    DataStax JIRA | 3 years ago | Joaquin Casares
    java.lang.IllegalStateException: 127.0.1.2 is not DOWN after 250s
  2. Speed up your debug routine!

    Automated exception search integrated into your IDE

  3. 0

    https://github.com/joaquincasares/java-driver/commit/a2a3f8890fc32ce4009c7889c90f34eccee48c14 This is unlike the issues that we've had before. Since the last issue was patched everything was working fine, aside from the race conditions that would last at most a few seconds. This new issue is that the waitFor* code can take as long as 250+ seconds before a node comes online. A 200 second delay handled a large number of cases, yet a 250 second delay handles all but typically 1 or 2. This also impacts the overall time it takes for tests to complete. The following issues tend to occur within the Reconnection and Retry Policy tests due to this new increase as well: {CODE} constantReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 78.054 sec <<< FAILURE! java.lang.AssertionError: Waited 70 seconds instead an expected 40 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:161) at com.datastax.driver.core.ReconnectionPolicyTest.constantReconnectionPolicyTest(ReconnectionPolicyTest.java:119) exponentialReconnectionPolicyTest(com.datastax.driver.core.ReconnectionPolicyTest) Time elapsed: 70.081 sec <<< FAILURE! java.lang.AssertionError: Waited 62 seconds instead an expected 30 seconds wait expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at com.datastax.driver.core.ReconnectionPolicyTest.reconnectionPolicyTest(ReconnectionPolicyTest.java:161) at com.datastax.driver.core.ReconnectionPolicyTest.exponentialReconnectionPolicyTest(ReconnectionPolicyTest.java:82) defaultLoggingPolicy(com.datastax.driver.core.RetryPolicyTest) Time elapsed: 430.156 sec <<< FAILURE! java.lang.IllegalStateException: 127.0.1.2 is not DOWN after 250s at com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:337) at com.datastax.driver.core.TestUtils.waitForDownWithWait(TestUtils.java:275) at com.datastax.driver.core.RetryPolicyTest.defaultPolicyTest(RetryPolicyTest.java:219) at com.datastax.driver.core.RetryPolicyTest.defaultLoggingPolicy(RetryPolicyTest.java:76) {CODE}

    DataStax JIRA | 3 years ago | Joaquin Casares
    java.lang.IllegalStateException: 127.0.1.2 is not DOWN after 250s
  4. 0

    Re: Forced to use Solution Set in Step Function

    flink-user | 2 years ago | Aljoscha Krettek
    java.lang.IllegalStateException: This stub is not part of an iteration step function.

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalStateException

      127.0.1.2 is not DOWN and is still part of the cluster after 20s

      at com.datastax.driver.core.TestUtils.waitFor()
    2. DataStax Java Driver for Apache Cassandra - Core
      LoadBalancingPolicyTest.tokenAwareTest
      1. com.datastax.driver.core.TestUtils.waitFor(TestUtils.java:338)
      2. com.datastax.driver.core.TestUtils.waitForDown(TestUtils.java:299)
      3. com.datastax.driver.core.LoadBalancingPolicyTest.tokenAwareTest(LoadBalancingPolicyTest.java:286)
      4. com.datastax.driver.core.LoadBalancingPolicyTest.tokenAwareTest(LoadBalancingPolicyTest.java:163)
      4 frames