java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace.

Apache's JIRA Issue Tracker | Tomasz Skorka | 2 years ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    AMQ 5.9 gets stuck under 30-50 req/second load when using JDBC persistence - this affects our application as it hangs during performance testing (this happens almost every night). Following stacktraces indicate that there's a deadlock on DB connection: Stack logged by C3P0, showing when first DB connection has been picked from the pool: {noformat} 2014-10-06 08:44:40,646 | INFO | Logging the stack trace by which the overdue resource was checked-out. | com.mchange.v2.resourcepool.BasicResourcePool | C3P0PooledConnectionPoolManager[identityToken->2x1e6s941raztn6mju110 java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace. at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:555) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:756) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:683) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) at org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:163) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:510) at org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:92) at org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:259) at org.apache.activemq.transaction.XATransaction.storeCommit(XATransaction.java:85) at org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:75) at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253) at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112) at org.apache.activemq.broker.TransportConnection.processCommitTransactionTwoPhase(TransportConnection.java:433) at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:102) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149) at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:662) {noformat} Following stack shows the same thread pending for second DB connection (without releasing the first one): {noformat} "ActiveMQ Transport: tcp:///10.132.7.20:36431@5445" daemon prio=10 tid=0x000000004119d000 nid=0x61bf in Object.wait() [0x00007f41120d7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000fabdd2c0> (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1414) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:606) - locked <0x00000000fabdd2c0> (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:526) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:756) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:683) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:290) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:840) at org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) at org.apache.activemq.store.memory.MemoryTransactionStore$4.run(MemoryTransactionStore.java:348) at org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:103) at org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:259) at org.apache.activemq.transaction.XATransaction.storeCommit(XATransaction.java:85) at org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:69) at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253) at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112) at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:424) at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149) at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:662) {noformat} Problem seems to be related with JDBCMessageStore.removeMessage method: {code:java} public void removeMessage(ConnectionContext context, MessageAck ack) throws IOException { long seq = persistenceAdapter.getStoreSequenceIdForMessageId(ack.getLastMessageId(), destination)[0]; ... {code} Call to {{removeMessage}} already has one DB connection passed in {{context}} method parameter, but calling {{persistenceAdapter.getStoreSequenceIdForMessageId}} creates another DB connection in the same transaction. Deadlock occurs when all DB connections are used by {{context}}, so that {{removeMessage}} can't fetch its own connection. Possible solution would be to pass {{ConnectionContext}} object to {{persistenceAdapter.getStoreSequenceIdForMessageId}} method, so that the method would reuse same connection.

    Apache's JIRA Issue Tracker | 2 years ago | Tomasz Skorka
    java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace.
  2. 0

    AMQ 5.9 gets stuck under 30-50 req/second load when using JDBC persistence - this affects our application as it hangs during performance testing (this happens almost every night). Following stacktraces indicate that there's a deadlock on DB connection: Stack logged by C3P0, showing when first DB connection has been picked from the pool: {noformat} 2014-10-06 08:44:40,646 | INFO | Logging the stack trace by which the overdue resource was checked-out. | com.mchange.v2.resourcepool.BasicResourcePool | C3P0PooledConnectionPoolManager[identityToken->2x1e6s941raztn6mju110 java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace. at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:555) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:756) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:683) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) at org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:163) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:510) at org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:92) at org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:259) at org.apache.activemq.transaction.XATransaction.storeCommit(XATransaction.java:85) at org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:75) at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253) at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112) at org.apache.activemq.broker.TransportConnection.processCommitTransactionTwoPhase(TransportConnection.java:433) at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:102) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149) at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:662) {noformat} Following stack shows the same thread pending for second DB connection (without releasing the first one): {noformat} "ActiveMQ Transport: tcp:///10.132.7.20:36431@5445" daemon prio=10 tid=0x000000004119d000 nid=0x61bf in Object.wait() [0x00007f41120d7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000fabdd2c0> (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1414) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:606) - locked <0x00000000fabdd2c0> (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:526) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:756) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:683) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:290) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:840) at org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) at org.apache.activemq.store.memory.MemoryTransactionStore$4.run(MemoryTransactionStore.java:348) at org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:103) at org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:259) at org.apache.activemq.transaction.XATransaction.storeCommit(XATransaction.java:85) at org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:69) at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253) at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112) at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:424) at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149) at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:662) {noformat} Problem seems to be related with JDBCMessageStore.removeMessage method: {code:java} public void removeMessage(ConnectionContext context, MessageAck ack) throws IOException { long seq = persistenceAdapter.getStoreSequenceIdForMessageId(ack.getLastMessageId(), destination)[0]; ... {code} Call to {{removeMessage}} already has one DB connection passed in {{context}} method parameter, but calling {{persistenceAdapter.getStoreSequenceIdForMessageId}} creates another DB connection in the same transaction. Deadlock occurs when all DB connections are used by {{context}}, so that {{removeMessage}} can't fetch its own connection. Possible solution would be to pass {{ConnectionContext}} object to {{persistenceAdapter.getStoreSequenceIdForMessageId}} method, so that the method would reuse same connection.

    Apache's JIRA Issue Tracker | 2 years ago | Tomasz Skorka
    java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace.
  3. 0

    c3p0 connection is overdue

    Stack Overflow | 2 years ago
    java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hibernate using C3P0 Works fine on Oracle but does not close connection with MySQL 5.6

    Stack Overflow | 3 years ago | Mithun
    java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace.
  6. 0

    What means this hibernate message "A checked-out resource is overdue"?

    Stack Overflow | 3 years ago | vs06
    java.lang.Exception: DEBUG STACK TRACE: Overdue resource check-out stack trace.

    3 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.Exception

      DEBUG STACK TRACE: Overdue resource check-out stack trace.

      at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource()
    2. c3p0:JDBC DataSources/Resource Pools
      AbstractPoolBackedDataSource.getConnection
      1. com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:555)
      2. com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:756)
      3. com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:683)
      4. com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
      4 frames
    3. ActiveMQ :: Core
      JDBCPersistenceAdapter.beginTransaction
      1. org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58)
      2. org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:163)
      3. org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:510)
      3 frames
    4. ActiveMQ :: Broker
      TransportConnection.processCommitTransactionTwoPhase
      1. org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:92)
      2. org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:259)
      3. org.apache.activemq.transaction.XATransaction.storeCommit(XATransaction.java:85)
      4. org.apache.activemq.transaction.XATransaction.commit(XATransaction.java:75)
      5. org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253)
      6. org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112)
      7. org.apache.activemq.broker.TransportConnection.processCommitTransactionTwoPhase(TransportConnection.java:433)
      7 frames
    5. ActiveMQ :: Core
      TransactionInfo.visit
      1. org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:102)
      1 frame
    6. ActiveMQ :: Broker
      TransportConnection$1.onCommand
      1. org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
      2. org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
      2 frames
    7. AMQ HTTP
      TransportSupport.doConsume
      1. org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
      2. org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
      3. org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
      4. org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      4 frames
    8. ActiveMQ :: Core
      TcpTransport.run
      1. org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
      2. org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
      2 frames
    9. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:662)
      1 frame