java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata

DataStax JIRA | Vishy Kasar | 2 years ago
  1. 0

    [JAVA-546] IllegalArgumentException: rpc_address is not a column defined in this metadata - DataStax

    atlassian.net | 1 year ago
    java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata
  2. 0

    Server limits the number of connections to 5. Client bombards the server over one host. At some point control connection is marked down and re-established. You get the following error: 2014-11-13 14:42:33 ERROR Cluster:1564 - Unknown error during reconnection to /ip1:9042, scheduling retry in 16000 milliseconds java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata at com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273) at com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279) at com.datastax.driver.core.ArrayBackedRow.getInet(ArrayBackedRow.java:239) at com.datastax.driver.core.ControlConnection.refreshNodeInfo(ControlConnection.java:438) at com.datastax.driver.core.Cluster$Manager$5.onReconnection(Cluster.java:1540) at com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124) The client code is shown below: PlainTextAuthProvider authProvider = new PlainTextAuthProvider("user", "pwd"); Collection<InetSocketAddress> whiteList = new ArrayList<InetSocketAddress>(); whiteList.add(new InetSocketAddress("ip1", 9042)); WhiteListPolicy policy = new WhiteListPolicy(new DCAwareRoundRobinPolicy("DC1"), whiteList); Cluster cluster = Cluster.builder().withAuthProvider(authProvider).withLoadBalancingPolicy(policy).addContactPoint("1ip1").build(); cluster.getConfiguration().getPoolingOptions().setMinSimultaneousRequestsPerConnectionThreshold(HostDistance.LOCAL, 0); cluster.getConfiguration().getPoolingOptions().setMaxSimultaneousRequestsPerConnectionThreshold(HostDistance.LOCAL, 1); cluster.getConfiguration().getPoolingOptions().setMaxConnectionsPerHost(HostDistance.LOCAL, 100); cluster.getConfiguration().getPoolingOptions().setCoreConnectionsPerHost(HostDistance.LOCAL, 3); final Session session = cluster.connect("system"); ExecutorService executorService = Executors.newFixedThreadPool(10); for (int i=0;i<100;++i) { executorService.execute(new Runnable() { @Override public void run() { while (true) readSystemPeersAndAssert(session); } }); } Uninterruptibles.sleepUninterruptibly(1, TimeUnit.HOURS); Full trace is attached.

    DataStax JIRA | 2 years ago | Vishy Kasar
    java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata
  3. 0

    Server limits the number of connections to 5. Client bombards the server over one host. At some point control connection is marked down and re-established. You get the following error: 2014-11-13 14:42:33 ERROR Cluster:1564 - Unknown error during reconnection to /ip1:9042, scheduling retry in 16000 milliseconds java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata at com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273) at com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279) at com.datastax.driver.core.ArrayBackedRow.getInet(ArrayBackedRow.java:239) at com.datastax.driver.core.ControlConnection.refreshNodeInfo(ControlConnection.java:438) at com.datastax.driver.core.Cluster$Manager$5.onReconnection(Cluster.java:1540) at com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124) The client code is shown below: PlainTextAuthProvider authProvider = new PlainTextAuthProvider("user", "pwd"); Collection<InetSocketAddress> whiteList = new ArrayList<InetSocketAddress>(); whiteList.add(new InetSocketAddress("ip1", 9042)); WhiteListPolicy policy = new WhiteListPolicy(new DCAwareRoundRobinPolicy("DC1"), whiteList); Cluster cluster = Cluster.builder().withAuthProvider(authProvider).withLoadBalancingPolicy(policy).addContactPoint("1ip1").build(); cluster.getConfiguration().getPoolingOptions().setMinSimultaneousRequestsPerConnectionThreshold(HostDistance.LOCAL, 0); cluster.getConfiguration().getPoolingOptions().setMaxSimultaneousRequestsPerConnectionThreshold(HostDistance.LOCAL, 1); cluster.getConfiguration().getPoolingOptions().setMaxConnectionsPerHost(HostDistance.LOCAL, 100); cluster.getConfiguration().getPoolingOptions().setCoreConnectionsPerHost(HostDistance.LOCAL, 3); final Session session = cluster.connect("system"); ExecutorService executorService = Executors.newFixedThreadPool(10); for (int i=0;i<100;++i) { executorService.execute(new Runnable() { @Override public void run() { while (true) readSystemPeersAndAssert(session); } }); } Uninterruptibles.sleepUninterruptibly(1, TimeUnit.HOURS); Full trace is attached.

    DataStax JIRA | 2 years ago | Vishy Kasar
    java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Produced an interesting case where the control connection cannot reestablish a Host that is the same host used for the control connection. The reason why it cannot be established is because ControlConnection#fetchNodeInfo makes the following query if the passed in Host is the same host used for the control connection: {code} SELECT * FROM system.local WHERE key='local' {code} The problem is that ControlConnection#refreshNodeInfo null checks the 'rpc_address' column of the returned row to determine if the node is an availble peer. The system.local table does not return the RPC address so this throws an IllegalArgumentException (added some extra comments in to demonstrate rpc_address is missing and that the control connection used the given host): {noformat} 26103 [Reconnection-1] DEBUG com.datastax.driver.core.Connection - Connection[/127.0.0.1:9042-69, inFlight=0, closed=false] Transport initialized and ready 26103 [Reconnection-1] DEBUG com.datastax.driver.core.Connection - Connection[/127.0.0.1:9042-69, inFlight=0, closed=true] closing connection 26103 [Reconnection-1] DEBUG com.datastax.driver.core.ControlConnection - [Control connection] Refreshing node info on /127.0.0.1:9042 26103 [Reconnection-1] WARN com.datastax.driver.core.ControlConnection - Host /127.0.0.1 is connected?: true, listenAddress null . 26109 [Reconnection-1] ERROR com.datastax.driver.core.ColumnDefinitions - Available indexes: {truncated_at=[I@3d748237, bootstrapped=[I@123e7724, gossip_generation=[I@42ffe5d7, partitioner=[I@2e4aa89e, rack=[I@278434d0, schema_version=[I@58a757f4, thrift_version=[I@3118604c, cluster_name=[I@375a36c2, host_id=[I@3c116f2, cql_version=[I@79c96bca, tokens=[I@16c3d93d, data_center=[I@3993e97f, key=[I@1dc6ae61, release_version=[I@5ee2718c, native_protocol_version=[I@648ccefe} 26109 [Reconnection-1] ERROR com.datastax.driver.core.Cluster - Unknown error during reconnection to /127.0.0.1:9042, scheduling retry in 8000 milliseconds java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata at com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:278) at com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:285) at com.datastax.driver.core.ArrayBackedRow.getInet(ArrayBackedRow.java:239) at com.datastax.driver.core.ControlConnection.refreshNodeInfo(ControlConnection.java:447) at com.datastax.driver.core.Cluster$Manager$5.onReconnection(Cluster.java:1595) at com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {noformat} I reproduced this by running a single node cassandra cluster running on a local interface. I executed a stress scenario and then used [tcpkill|http://linux.die.net/man/8/tcpkill] to reset connections on port 9042 and then terminated tcpkill shortly after to allow connections to reestablish: {code} sudo /usr/local/sbin/tcpkill -i lo0 port 9042 {code}

    DataStax JIRA | 2 years ago | Andy Tolbert
    java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata
  6. 0

    Produced an interesting case where the control connection cannot reestablish a Host that is the same host used for the control connection. The reason why it cannot be established is because ControlConnection#fetchNodeInfo makes the following query if the passed in Host is the same host used for the control connection: {code} SELECT * FROM system.local WHERE key='local' {code} The problem is that ControlConnection#refreshNodeInfo null checks the 'rpc_address' column of the returned row to determine if the node is an availble peer. The system.local table does not return the RPC address so this throws an IllegalArgumentException (added some extra comments in to demonstrate rpc_address is missing and that the control connection used the given host): {noformat} 26103 [Reconnection-1] DEBUG com.datastax.driver.core.Connection - Connection[/127.0.0.1:9042-69, inFlight=0, closed=false] Transport initialized and ready 26103 [Reconnection-1] DEBUG com.datastax.driver.core.Connection - Connection[/127.0.0.1:9042-69, inFlight=0, closed=true] closing connection 26103 [Reconnection-1] DEBUG com.datastax.driver.core.ControlConnection - [Control connection] Refreshing node info on /127.0.0.1:9042 26103 [Reconnection-1] WARN com.datastax.driver.core.ControlConnection - Host /127.0.0.1 is connected?: true, listenAddress null . 26109 [Reconnection-1] ERROR com.datastax.driver.core.ColumnDefinitions - Available indexes: {truncated_at=[I@3d748237, bootstrapped=[I@123e7724, gossip_generation=[I@42ffe5d7, partitioner=[I@2e4aa89e, rack=[I@278434d0, schema_version=[I@58a757f4, thrift_version=[I@3118604c, cluster_name=[I@375a36c2, host_id=[I@3c116f2, cql_version=[I@79c96bca, tokens=[I@16c3d93d, data_center=[I@3993e97f, key=[I@1dc6ae61, release_version=[I@5ee2718c, native_protocol_version=[I@648ccefe} 26109 [Reconnection-1] ERROR com.datastax.driver.core.Cluster - Unknown error during reconnection to /127.0.0.1:9042, scheduling retry in 8000 milliseconds java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata at com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:278) at com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:285) at com.datastax.driver.core.ArrayBackedRow.getInet(ArrayBackedRow.java:239) at com.datastax.driver.core.ControlConnection.refreshNodeInfo(ControlConnection.java:447) at com.datastax.driver.core.Cluster$Manager$5.onReconnection(Cluster.java:1595) at com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {noformat} I reproduced this by running a single node cassandra cluster running on a local interface. I executed a stress scenario and then used [tcpkill|http://linux.die.net/man/8/tcpkill] to reset connections on port 9042 and then terminated tcpkill shortly after to allow connections to reestablish: {code} sudo /usr/local/sbin/tcpkill -i lo0 port 9042 {code}

    DataStax JIRA | 2 years ago | Andy Tolbert
    java.lang.IllegalArgumentException: rpc_address is not a column defined in this metadata

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      rpc_address is not a column defined in this metadata

      at com.datastax.driver.core.ColumnDefinitions.getAllIdx()
    2. DataStax Java Driver for Apache Cassandra - Core
      AbstractReconnectionHandler.run
      1. com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
      2. com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
      3. com.datastax.driver.core.ArrayBackedRow.getInet(ArrayBackedRow.java:239)
      4. com.datastax.driver.core.ControlConnection.refreshNodeInfo(ControlConnection.java:438)
      5. com.datastax.driver.core.Cluster$Manager$5.onReconnection(Cluster.java:1540)
      6. com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124)
      6 frames