org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction::48:local. Lock is held by GlobalTransaction::45:remote at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

  1. 0

    wildfly 10 + HA (infinispan) + websocket locking issues

    Stack Overflow | 4 months ago | Yannick Menager
    org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction::48:local. Lock is held by GlobalTransaction::45:remote at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
  2. 0

    Seen in these scenarios in ER3: - http-session-jvmkill-dist-async-3owners - ejb-ejbservlet-shutdown-dist-async Immediately after perf18 was killed (jvmkill), perf19 and perf20 logged JBEAP-794 and then this error: {code} [0m[31m04:21:28,668 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-127) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 9 [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92) [JBossINF] at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:259) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:103) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [JBossINF] at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828) [JBossINF] at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:82) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:67) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370) [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231) [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) {code} This one followed right after: {code} [JBossINF] [0m[31m04:21:28,717 ERROR [io.undertow.request] (default task-127) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 9 [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92) [JBossINF] at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:259) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:103) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [JBossINF] at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828) [JBossINF] at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:82) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:67) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370) [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231) [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) {code} The stacktrace is different from JBEAP-735 although it can be related. Link: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-session-jvmkill-dist-async-3owners/15/console-perf19/

    JBoss Issue Tracker | 12 months ago | Michal Vinkler
    org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 9
  3. 0

    GitHub comment 8761#196427507

    GitHub | 9 months ago | wildfly-ci
    org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key UnknownSessionID [5766555655704850565248495257505466495652574965675369695269575550] and requestor GlobalTransaction:<null>:367:local. Lock is held by GlobalTransaction:<null>:366:local
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Deploying on wildfly produces "Unable to acquire lock after 15 seconds" exception

    Stack Overflow | 1 year ago | Gr8 Java
    org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key qDV8eCOJ_Dwt1aDZAMtOELUylOE7XNTN0IpHi34V and requestor GlobalTransaction:<null>:20:local. Lock is held by GlobalTransaction:<null>:19:local
  6. 0

    Deploying on wildfly produces "Unable to acquire lock after 15 seconds" exception

    Stack Overflow | 1 year ago | Gr8 Java
    org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key qDV8eCOJ_Dwt1aDZAMtOELUylOE7XNTN0IpHi34V and requestor GlobalTransaction:<null>:20:local. Lock is held by GlobalTransaction:<null>:19:local

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.infinispan.util.concurrent.TimeoutException

      ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction::48:local. Lock is held by GlobalTransaction::45:remote at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord()
    2. Infinispan Core
      AbstractDelegatingAdvancedCache.lock
      1. org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)
      2. org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)
      3. org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)
      4. org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)
      5. org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
      6. org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      7. org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
      8. org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)
      9. org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
      10. org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      11. org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
      12. org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
      13. org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)
      14. org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
      15. org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      16. org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)
      17. org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)
      18. org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
      19. org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      20. org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
      21. org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)
      22. org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
      23. org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      24. org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
      25. org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
      26. org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
      27. org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
      28. org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)
      29. org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)
      30. org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)
      30 frames
    3. org.wildfly.clustering
      InfinispanSessionManager.findSession
      1. org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)
      2. org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)
      3. org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
      4. org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
      5. org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
      6. org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
      6 frames
    4. WildFly: Web session clustering - Undertow integration
      DistributableSessionManager.getSession
      1. org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
      1 frame
    5. Undertow Servlet
      CachedAuthenticatedSessionHandler.handleRequest
      1. io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)
      2. io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)
      3. io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)
      3 frames
    6. Undertow Core
      PredicateHandler.handleRequest
      1. io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      2. io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      3. io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      3 frames
    7. WildFly: Undertow
      JACCContextIdHandler.handleRequest
      1. org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      1 frame
    8. Undertow Core
      PredicateHandler.handleRequest
      1. io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      2. io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      2 frames
    9. Undertow Servlet
      ServletInitialHandler$1.handleRequest
      1. io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
      2. io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
      3. io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      4. io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
      4 frames
    10. Undertow Core
      HttpServerExchange$1.run
      1. io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
      2. io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
      2 frames
    11. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      3. java.lang.Thread.run(Thread.java:745)
      3 frames