com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types

DataStax JIRA | Rafi Shamim | 12 months ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    I turned this into a ticket after being suggested to do so in this [java-driver-user list post|https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/GIz1IWJDVNQ]. I ran into a problem where occasionally, under somewhat heavy load, most or all instances of my app would lose connections to nodes in the Cassandra cluster. After turning on debug logs, I found that it was because connections were getting defuncted: {noformat} c.d.driver.core.Host.STATES : Defuncting Connection[/172.23.31.8:9042-19, inFlight=0, closed=false] because: [/172.23.31.8:9042] Unexpected exception triggered (com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types) {noformat} Earlier in the logs, the InvalidTypeException comes from the query logger. It failed when trying to format a CustomType into a String. {noformat} c.d.driver.core.Connection : Connection[/172.23.31.8:9042-19, inFlight=0, closed=false] connection error com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types at com.datastax.driver.core.DataType$Custom.format(DataType.java:910) at com.datastax.driver.core.QueryLogger.parameterValueAsString(QueryLogger.java:753) at com.datastax.driver.core.QueryLogger.appendParameters(QueryLogger.java:723) at com.datastax.driver.core.QueryLogger.logQuery(QueryLogger.java:650) at com.datastax.driver.core.QueryLogger$ConstantThresholdQueryLogger.maybeLogSlowQuery(QueryLogger.java:269) at com.datastax.driver.core.QueryLogger$ConstantThresholdQueryLogger.maybeLogNormalOrSlowQuery(QueryLogger.java:260) at com.datastax.driver.core.QueryLogger.update(QueryLogger.java:623) at com.datastax.driver.core.Cluster$Manager.reportLatency(Cluster.java:1597) at ... {noformat} It turns out that what was happening was: - Under moderate/heavy load, the driver would try to log slow queries (I had slow query logging on, and normal query logging off). - The logging would fail with that exception, which would defunct the connection, and cause the request to fail. - The request would then get retried on another host and that would likely fail too, causing a feedback loop until the load went down enough for slow queries to stop getting logged. - At some point, during some of these incidents, all connections to Cassandra might have been closed simultaneously, meaning my app could not talk to the database at all until reconnection happened. My table schema is rather old, and it still uses the LexicalUUIDType, which is a CustomType. I can't easily switch to UUIDType since it is the partition key. For now, as a workaround, I turned off all query logging. Can this be fixed so that query logging can be used with CustomTypes? It seems like a bug that an issue with formatting a string for a log statement would cause connections to get dropped. Note that version 3.0 of the driver doesn't seem to have this issue, since it can format CustomTypes into a String without error. I think in version 2.1 the exception could get handled better. Or the {{DataType$Custom.format}} method could use {{Bytes.toHexString}} (as version 3.0 of the driver does) instead of throwing InvalidTypeException.

    DataStax JIRA | 12 months ago | Rafi Shamim
    com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types
  2. 0

    I turned this into a ticket after being suggested to do so in this [java-driver-user list post|https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/GIz1IWJDVNQ]. I ran into a problem where occasionally, under somewhat heavy load, most or all instances of my app would lose connections to nodes in the Cassandra cluster. After turning on debug logs, I found that it was because connections were getting defuncted: {noformat} c.d.driver.core.Host.STATES : Defuncting Connection[/172.23.31.8:9042-19, inFlight=0, closed=false] because: [/172.23.31.8:9042] Unexpected exception triggered (com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types) {noformat} Earlier in the logs, the InvalidTypeException comes from the query logger. It failed when trying to format a CustomType into a String. {noformat} c.d.driver.core.Connection : Connection[/172.23.31.8:9042-19, inFlight=0, closed=false] connection error com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types at com.datastax.driver.core.DataType$Custom.format(DataType.java:910) at com.datastax.driver.core.QueryLogger.parameterValueAsString(QueryLogger.java:753) at com.datastax.driver.core.QueryLogger.appendParameters(QueryLogger.java:723) at com.datastax.driver.core.QueryLogger.logQuery(QueryLogger.java:650) at com.datastax.driver.core.QueryLogger$ConstantThresholdQueryLogger.maybeLogSlowQuery(QueryLogger.java:269) at com.datastax.driver.core.QueryLogger$ConstantThresholdQueryLogger.maybeLogNormalOrSlowQuery(QueryLogger.java:260) at com.datastax.driver.core.QueryLogger.update(QueryLogger.java:623) at com.datastax.driver.core.Cluster$Manager.reportLatency(Cluster.java:1597) at ... {noformat} It turns out that what was happening was: - Under moderate/heavy load, the driver would try to log slow queries (I had slow query logging on, and normal query logging off). - The logging would fail with that exception, which would defunct the connection, and cause the request to fail. - The request would then get retried on another host and that would likely fail too, causing a feedback loop until the load went down enough for slow queries to stop getting logged. - At some point, during some of these incidents, all connections to Cassandra might have been closed simultaneously, meaning my app could not talk to the database at all until reconnection happened. My table schema is rather old, and it still uses the LexicalUUIDType, which is a CustomType. I can't easily switch to UUIDType since it is the partition key. For now, as a workaround, I turned off all query logging. Can this be fixed so that query logging can be used with CustomTypes? It seems like a bug that an issue with formatting a string for a log statement would cause connections to get dropped. Note that version 3.0 of the driver doesn't seem to have this issue, since it can format CustomTypes into a String without error. I think in version 2.1 the exception could get handled better. Or the {{DataType$Custom.format}} method could use {{Bytes.toHexString}} (as version 3.0 of the driver does) instead of throwing InvalidTypeException.

    DataStax JIRA | 12 months ago | Rafi Shamim
    com.datastax.driver.core.exceptions.InvalidTypeException: Cannot format values of custom types

    Root Cause Analysis

    1. com.datastax.driver.core.exceptions.InvalidTypeException

      Cannot format values of custom types

      at com.datastax.driver.core.DataType$Custom.format()
    2. DataStax Java Driver for Apache Cassandra - Core
      Cluster$Manager.reportLatency
      1. com.datastax.driver.core.DataType$Custom.format(DataType.java:910)
      2. com.datastax.driver.core.QueryLogger.parameterValueAsString(QueryLogger.java:753)
      3. com.datastax.driver.core.QueryLogger.appendParameters(QueryLogger.java:723)
      4. com.datastax.driver.core.QueryLogger.logQuery(QueryLogger.java:650)
      5. com.datastax.driver.core.QueryLogger$ConstantThresholdQueryLogger.maybeLogSlowQuery(QueryLogger.java:269)
      6. com.datastax.driver.core.QueryLogger$ConstantThresholdQueryLogger.maybeLogNormalOrSlowQuery(QueryLogger.java:260)
      7. com.datastax.driver.core.QueryLogger.update(QueryLogger.java:623)
      8. com.datastax.driver.core.Cluster$Manager.reportLatency(Cluster.java:1597)
      8 frames